If you compile Python, you may also try nuitka, which can not only compile a standalone program, but also compile a single package.
You Should Compile Your Python and Here’s Why
61–70 of 109 posts
Re: You Should Compile Your Python and Here’s Why
#62I've been writing some python the last few days. Today I did something terrible. I like to believe it was out of wisdom rather than ignorance. I needed to do some nuanced bit manipulations, and so I built up a string of ascii 0's and 1's, sliced the string, and then converted back to integers. I worked on a codebase many years ago when I was an intern. That code would read frames off a CAN bus in a car. For some reas…
> I built up a string of ascii 0's and 1's, sliced the string, and then converted back to integers. How about a list of integers: $ txr This is the TXR Lisp interactive listener of TXR 274. Quit with :quit or Ctrl-D on an empty line. Ctrl-X ? for cheatsheet. This could be the year of the TXR desktop; I can feel it! 1> (digits 37) (3 7) 2> (digits 37 2) (1 0 0 1 0 1) 3> (reverse *2) (1 0 1 0 0 1) 4> (mapcar (op - 1) *…
No post body was provided.
Re: You Should Compile Your Python and Here’s Why
#63I'm learning Python. Do someone knows a resource to learn the well-known tricks for "fast" Python? A book, a website, a cheat sheet or even a MOOC or part of a MOOC?
No post body was provided.
Re: You Should Compile Your Python and Here’s Why
#64Earlier quoted context omitted.
I don’t always know how fast will be fast enough when I start a project and the risk of needing a rewrite really puts me off Python.
This is premature optimization right
I don’t believe so. Choosing a language that is faster (and still productive) from the get-go has almost no cost but reduces the risk of rewrite.
Re: You Should Compile Your Python and Here’s Why
#65Not to be shit-eating, but I never understood why people use python instead of Go other than for ML teams. I program mostly in Java and Go so would love a perspective.
Python is incredibly easy to learn, so more people learn it as their first language these days.
Re: You Should Compile Your Python and Here’s Why
#66I wish I can compile pandas
Try looking into polars as a much faster pandas alternative.
Re: You Should Compile Your Python and Here’s Why
#67I'm learning Python. Do someone knows a resource to learn the well-known tricks for "fast" Python? A book, a website, a cheat sheet or even a MOOC or part of a MOOC?
There are plenty of things for which python is not slow, but people using python don't know a lot about programming, and end up with a slow result.
A couple of advises:
- the right algo will go a long, long way.
- know your data structures. E.G: assigning to a slice is ridiculously fast (even while unpacking), memory views may save a lot on byte heavy workloads, heapq and deque are underrated, etc. Also check out https://wiki.python.org/moin/TimeComplexity for big O notations on python builtin types common operations to understand what you pay for.
- know the stdlib. collections, itertools and functools all contain incredible gems.
- delegate. Python is a fantastic glue language, use it for what it's good at. Your database, your numpy arrays, your cache are all amazing are what they do, no matter the language. Let them do the heavy work.
- don't kill good perfs by ignorance. I regularly see people casting a generator, iterating on a dataframe, calling readlines() on a file or doing something else that is destroying the otherwise excellent perfs of their program.
- know the ecosystem. There are some very good fast libs out there: diskcache, sortedcontainer, scipy, uvloop...
- use threads to avoid blocking a GUI, multiprocess to share work between CPU and asyncio to speed up network operations. Each tool has a sweet spot. But threads are underrated, they work well for a couple of hundred parallel network operations, and most C libs will actually release the GIL, so they can use several CPU more often than you'd thin. Also, use pools if you can, shared_memory in 3.8 or mmap.
- sometime the dirty solution is just faster, like subprocessing to ffmpeg.
- The more recent, the slower. Sure, I love statistics, pathlib and dataclasses. In a regular code they are great. On a bottleneck however, they are very slow.
- the array module is not supposed to speed up the code, only save memory. But sometimes it does.
- printing to the terminal is limiting. Sometime your program is doing fine, the display is preventing it to go faster. At least check the flush.
- comprehensions are faster than alternatives.
- pre-allocating lists and dicts can help.
- measure. The austin profiler is your friend.
- rewriting the hot path in a faster language is likely more interesting than writing the whole program in it. A bit of nim or rust is easy to call from python.
- some python distribution are faster than others. I don't mean just pypy, but also regular cpyhon that have been built specially for some architectures. E.G: https://www.intel.com/content/www/us/en/developer/tools/onea...
- nuitka code compilation can speed up the code by a factor of 4, it's nice, especially for startup.
Re: You Should Compile Your Python and Here’s Why
#68Earlier quoted context omitted.
Probably for the same reasons people use Java and Go instead of C. It's just more convenient sometimes. But sometimes not. In terms of building a team at a company? In my experience it's been arbitrary. Manager / Lead has experience in language X, decides to hire people who also know it. Or all the other teams are already using language X. Or Manager / Lead has heard "X language is good at Y" and decides to go with t…
Yeah I get that, the same reason why I question some of the teams I've been on choice of JS frameworks, it's just what the developers were familiar with. Go compiles so fast and reads cleaner imo, that I never saw the benefit of python. I always thought debugging python was clunky, could be unfamiliarity with interpreted languages. Also a few times I needed to use an ODBC driver with python left a bad taste in my mou…
[deleted]
Re: You Should Compile Your Python and Here’s Why
#69I'm learning Python. Do someone knows a resource to learn the well-known tricks for "fast" Python? A book, a website, a cheat sheet or even a MOOC or part of a MOOC?
There are plenty of things for which python is not slow, but people using python don't know a lot about programming, and end up with a slow result. A couple of advises: - the right algo will go a long, long way. - know your data structures. E.G: assigning to a slice is ridiculously fast (even while unpacking), memory views may save a lot on byte heavy workloads, heapq and deque are underrated, etc. Also check out htt…
A big thank you! This is an amazing list of things to look into and keep in mind. Many points are obscure to me, but I'm going to print this and keep it at hand. Thanks again.
Re: You Should Compile Your Python and Here’s Why
#70I've been typing SQLAlchemy for some months now and there's lots of scenarios where if you end up on mypy's issue tracker (which is very often), you will see "oh use a #type: ignore to deal with this occasional use case" as official advice. And here we see a compiler that considers all "# type: ignore" to be bugs and will eventually refuse to compile them. That's simply not realistic for the way Python typing works r…
mypyc has escape hatches, e.g. casting to Any is a common trick.