141–150 of 336 posts
The go-to solution for speeding up Python code should always be first to use Cython on critical sections of your Python code and tweak your code using type annotations, at least IMHO.
Do type annotations really make any difference to the interpreter? I thought that the interpreter doesn't care about what type a variable is annotated to...
I don't think it's true to say that Python's core developers are uninterested in performance. Speeding up Python is a hard problem. He mentions PyPy but even that has only managed modest performance gains in some areas (and not without tradeoffs). He suggests JavaScript as a comparison but doesn't elaborate on how they're comparable beyond the superficial (they're both dynamic scripting languages). I get that he's fr…
Python's little tin god really likes his CPython implementation being the One True Python. Python does the things that are easy to do in an interpreter where everything is a dictionary, and avoids things which are hard to do in that environment. In Python, you can store into any variable in any thread from any other thread. You can replace code being executed in another thread. Even Javascript doesn't let you do that…
No, he likes it to be the reference implementation, as it both is and should be. It's simple for a reason.
> This functionality is very rarely used
It's used all the time by debuggers, and the underlying features that allow you to do this is one of the most core and intrinsic things in Python.
I find this argument breaks down if you consider human psychology. Especially with a program taking 15 seconds or 30 minutes (which is a reasonable time span between say a C++/Rust implementation and a Python implementation in some cases I've experienced).
With 15 seconds exec time you might stay in flow. With 30 minutes you're almost guaranteed to have started something else. Maybe you even forget and only get back to it the next day. All of a sudden your 30minutes delay become a day. Then you notice you made some wrong inputs, and you lose another day. In the other case you're still under a minute.
I find small increases in program delay often lead to big increases in time inefficiency. It's hard to constantly context switch in and out of tasks.
I think the utilitarian argument should take human psychology into effect and weight more towards faster programs.
Earlier quoted context omitted.
I mean, if you're willing to give up the ecosystem, there are tons of options out there.
Exactly, the ecosystem is the only reason IMHO why Python is "easy". I find I'm much quicker at developing C#/F# if the library is available on NuGet (which is often, but not always the case). They're both reasonably fast, too. Python only has a very large community and thus ecosystem to offer.
No, Python was invented at a time when its closest competitor was Perl - and you need only compare typical Perl with typical Python to appreciate that Python really was a usability revelation.
But that was nearly 30 years ago. I do think we can do better now.
Interesting article. I mostly agree. OP, could I ask a question? You mention 1TB files. Why do you guys at embl not use a database for this sort of stuff? I'd figure that with some proper indexing, I figure you could see pretty decent speedups just from that already.
However, databases are hard to share so many steps require dumping the database into some interchange formats (custom and often from before the age of XML or JSON, yeah for ASN.1 parsing!)
Sharing database dumps is done but commercial licenses and version mismatches do add issues here as well. Remember EMBL/ENA is older than MySQL. The databases tend to have the wrong shape for the next downstream step i.e. table design is related to work flow and if your next step in working is completely different we end up with issues. Also some data can't be published until a certain date so that needs to be filtered from the dumps in some way.
Consider as well that this project is 3 decades old and used to be printed in books at some point, and shipped on DVD as recently as 2004. File based operations can be extremely efficient.
I don't find this a very compelling argument. The author doesn't mention any attempts to profile or speed up the code. Specifically with pandas I've found if you aren't careful you can do a lot of unnecessary copying. Not sure if that's what is going on here, but cProfile can help find the bottlenecks.
This is a weird article at this point in time. The question it addresses: "Does Python's performance matter?" Has always had the answer: "Sometimes, and you have options for those cases." The OP found a "sometimes", and he's using one of those options. In this case, he's got Python for prototyping and glue, with Haskell improving performance. This is as it should be. I don't know of any Python advocates who say it's…
I’d like to thank the author for sharing a very practical view of problem solving in the data science space. Can I suggest julia? Its very easy to understand coming from python, and performant code can be had usually in easy to read implementation of the expressions in whatever paper you are basing your work upon.
I have never been much of a fan of Python. 15 years ago at lunch Peter Norvig was talking about the advantages of Python (he and I wrote Common Lisp books at the same time). I then tried Python for a few months but then went back to Ruby for a scripting language and Java, Common Lisp, and Haskell for non-scripting tasks.
I now manage a machine learning team that is all-in using Python so that is what I am using also. I am starting to really like Python, and by using type annotations, heavy use of pylint and PyCharm, I am finding Python really nice to use. Spinning up on using Cython is on my want-to-do list also.
This is a weird article at this point in time. The question it addresses: "Does Python's performance matter?" Has always had the answer: "Sometimes, and you have options for those cases." The OP found a "sometimes", and he's using one of those options. In this case, he's got Python for prototyping and glue, with Haskell improving performance. This is as it should be. I don't know of any Python advocates who say it's…
In the last few years, there's been a lot of hype about replacing other number crunching solutions (R, SPSS, even Matlab) with the Python ecosystem of tools (Pandas, SciPy, etc.).
Earlier quoted context omitted.
GHC Haskell is an advanced optimising compiler, which can get very near the speed of C and C++. However, to write fast programs, one must use the right data structures and algorithms. Often this means array-based strings and streaming IO, unfortunately many Haskell textbooks don't tend to cover this.
All comparisons of Haskell and C I have seen where Haskell has been near the performance of C(++) has always compared a highly optimised Haskell program to a moderately fast c program. I have found that most managed languages generally come within 2-5 times slower than C and C++. Which is good enough for me.
cpp webserver + sqlite database + cygwin overhead? 9MB RAM lua worker + websocket client? 4MB RAM JVM server + static html page + websocket relay + kurento api? 500MB RAM
With the exception of the cpp webserver none of these tasks are CPU or memory intensive yet the JVM is still off by orders of magnitude.
It's ok if it's the only application running on a server with multiple users but there is just a single user and that's me.