171–180 of 336 posts
Earlier quoted context omitted.
If I can fit the code into numpy-like structure, then Python is typically fine. The issue is when I cannot.
Have you given dask a try? It gives you out-of-core arrays with numpy semantics and distributed computing.
If you can't make the core pandas code decently fast, dask won't save you.
Is writing extensions a lost art? I read a few blog posts about speeding up Python and Ruby with Rust extensions. This should enable rewriting only the slow parts. Later, you could replace more of it if needed. Is writing extensions so very problematic in practice? I know Go has runtime issues making it not very good for mixing with other languages, so it often encourages rewriting the whole application in it.
Extensions aren't a total solution, though, which people often sell them as. You have an impedance mismatch between Python and C code, because Python has all of its objects packed in a way that is very strange to C, so you end up essentially deserializing all objects into C, then back out into Python, in a very expensive and allocation-heavy (on both sides) conversion. If you can set up your computation in Python and…
The author claims that Python has the lowest developer cost. I used to think that was true, and maybe it is in data science applications, but I regularly find that I'm quite a bit more productive in Go than in Python (largely thanks to the type checker and other static analysis tooling). As an added bonus, Go programs are regularly 100 times faster than Python programs, and usually Python programs are much more difficult to optimize than Go programs.
Library availability notwithstanding, starting new projects (of any significance at all) in Python is looking like a worse and worse choice all the time.
Earlier quoted context omitted.
Interesting assertion re: TimeToWriteCode, but I think there's TimeToWriteCode vs. TimeToWriteGoodCode. I'm working on my first serious Python project right now, and I find it's super easy to throw together some code that more or less works; but for solid, readable, documented, properly unit-tested code I hope is production-ready, it's not any faster than Perl or Golang. (Sure, if you're a Python expert it's faster f…
Production-ready is so complex, it's hard to make any comparison. E.g. for a library, writing good documentation (with diagrams and decent technical writing) takes me way longer coding anyway - probably by an order of magnitude. Proper unit-testing is also going to take roughly the same time in any language, just because you have to think hard about sensible tests (although I still love mocking/patching in Python, so…
One big point I would give to Golang, about which lots of people disagree with me, is the "opinionatedness" of it. It seems to me that Python, like Perl, has a "There's More Than One Way To Do It" mentality, and after many years of that I really appreciated Golang's emphasis on the "idiomatic." That goes for the tooling too.
I have also noticed that the Python ecosystem doesn't have a strong documentation culture, which I find annoying as a relative newbie. But that presumably matters less over time, and it seems to be part of the Python Way to use libraries that "just work" and not worry about the details.
Earlier quoted context omitted.
From 4 years ago... a lot has changed since. Consider this recent post for example: https://tk3369.wordpress.com/2018/02/04/an-updated-analysis-...
The linked article doesn't attempt to refute the significant claims from Dan's article (which has an update from a year later, so 2015, at the bottom): 1. That the language is (was) undertested, and as a result, full of easy to run into bugs 2. The language makes it easy to ignore errors 3. APIs are inconsistent 4. The head branch isn't kept build-clean (i.e., often it fails to build) 5. Code is often undocumented, w…
For example, the base language now has 92% test coverage. CI is being run on Windows 32/64 bit, Linux 32/64bit, macOS, FreeBSD with ARM CI coming up. Every day a large number of benchmarks are run on dedicated hardware and the results are tracked. Benchmarks are also run on most PRs that has potential performance implications bhefore merging. Before new point releases, the tests for all registered packages are run and compared to the old version. Every new failure in a package test is tracked down to make sure there is nothing breaking in the point release.
While package load time is still an issue, the introduced precompilation feature has significantly helped with this. Also, new methods of working with the language, like Revise.jl (https://github.com/timholy/Revise.jl) which updates the code that is being executed in real time when you save your file, makes load time much less of an issue.
No matter who the author is of a blogpost, if it is made about something that has been in rapid development for years after the post was written, the information will have little relevance to the current situation.
Is writing extensions a lost art? I read a few blog posts about speeding up Python and Ruby with Rust extensions. This should enable rewriting only the slow parts. Later, you could replace more of it if needed. Is writing extensions so very problematic in practice? I know Go has runtime issues making it not very good for mixing with other languages, so it often encourages rewriting the whole application in it.
Extensions aren't a total solution, though, which people often sell them as. You have an impedance mismatch between Python and C code, because Python has all of its objects packed in a way that is very strange to C, so you end up essentially deserializing all objects into C, then back out into Python, in a very expensive and allocation-heavy (on both sides) conversion. If you can set up your computation in Python and…
Pyo3 library gives you ability to work both diractions. Call python code from rust and call rust code from python.
I love Python. The language is a joy, the eco system is fantastic. But yes, let’s be honest, if you can not vectorise your code it is slow, and I think that will be its downfall eventually. I’m excited about Julia, I hope it gains popularity and the eco system grows. Until then, and in particular until the data frames story can compete with pandas, it Python with Cython for me, but I’d rather skip the Cython if it wa…
Also I don't feel like the culture of Python programming focuses too much on documenting things which makes reading code at times like transcribing ancient Latin manuscripts. Maybe a good analogy would be JS back in the days with global jQuery scripts. Too unrestricted and free-form maybe. I'd wish Python became more like Kotlin with very clear patterns and great IDE support (in addition to PyCharm).
Well those are at least my experiences and feel free to disagree with me.
Earlier quoted context omitted.
Exactly, the ecosystem is the only reason IMHO why Python is "easy". I find I'm much quicker at developing C#/F# if the library is available on NuGet (which is often, but not always the case). They're both reasonably fast, too. Python only has a very large community and thus ecosystem to offer.
It can't be the only reason, or how would it ever have attracted such an ecosystem in the first place? Especially with its performance characteristics, and lack of corporate backing. No, Python was invented at a time when its closest competitor was Perl - and you need only compare typical Perl with typical Python to appreciate that Python really was a usability revelation. But that was nearly 30 years ago. I do think…
I don't find this a very compelling argument. The author doesn't mention any attempts to profile or speed up the code. Specifically with pandas I've found if you aren't careful you can do a lot of unnecessary copying. Not sure if that's what is going on here, but cProfile can help find the bottlenecks.
Well, I just wanted to use pandas to load a 4GB csv file. After using 32GB of my RAM, and 4GB of swap I gave up. I've just loaded all that data to Postgres, and made a couple of queries. This way I stopped using pandas at all.
After that I usually just keep the numpy array since all I need is floats. I guess the index groupby stuff is cool, but I never really needed it. Postgres is fine but if you're just doing numerics it doesn't help much.
Where are the main blowouts in python performance? For example, is it compilation, evaluation overhead, or memory management?
Naive interpretation of the bytecode (not even pre-decoded, just a switch statement). And almost everything is resolved in the dynamic environment. for example, a = foo.bar(b) is actually ldict = locals() ldict['a'] = ldict['foo'].__getattribute__('bar')(ldict['b'])
So your code should be more like:
locals[a_idx] = locals[foo_idx]->__getattribute__('bar')(locals[b_idx])
(Also, interpreter dispatch uses computed GOTOs instead of the plain switch on C compilers that support it.)