Obligatory reference to Hy / Hylang, a Lisp that compiles to the Python AST.
Python vs Common Lisp, workflow and ecosystem (2019)
21–30 of 83 posts
Re: Python vs Common Lisp, workflow and ecosystem (2019)
#22Earlier quoted context omitted.
> and is fast because the speedy parts aren't in Python. Having worked months with a slew of senior data scientists, this was a bit painful. Python is so slow and those data scientists were very good at coming up with solutions for the issues of the company, but the implementations (using Spacy, Pandas and other libs) had enough Python in them to make them not practical for the company use case. Nice prototypes which…
But that's fine, no? I mean, it's a pretty common workflow where the people close to the science part of something write a prototype in their language/ecosystem of choice, and then the engineering side is in charge of taking the prototype implementation and making it performant enough for production use. Finding people who know both, data science, and low level programming languages well enough to be able to implemen…
Yes, I guess it is fine if that is the flow. I just didn't expect it upfront (my bad).
Re: Python vs Common Lisp, workflow and ecosystem (2019)
#23Earlier quoted context omitted.
Yeah, they're very much doing it. Pandas is huge, libraries like Spacy, NetworkX, etc exist. It's a massive and good ecosystem. Python is the goto for scientific computing in most of the sciences for newer students I'd hazard a guess over the older R and Julia. This will be blindingly obvious if you work in that area. Yes, you can do it in another language, but you're missing out on a lot of stuff that is just done a…
> and is fast because the speedy parts aren't in Python. Having worked months with a slew of senior data scientists, this was a bit painful. Python is so slow and those data scientists were very good at coming up with solutions for the issues of the company, but the implementations (using Spacy, Pandas and other libs) had enough Python in them to make them not practical for the company use case. Nice prototypes which…
(I'm the creator of spaCy)
If you want to email me at matt@explosion.ai , I'd be interested in the specifics of the algorithm and why the implementation was slow.
The idea for something like that keyword extraction algorithm would be that if the Python API is slow, you should just use Cython. The Cython API of spaCy is really fast because the `Doc` is just a `TokenC*`, and the tokens just hold a pointer to their lexeme struct, which has the various attributes encoded as integers.
I've never really done a good job of teaching people to use the Cython API though. I completely agree that it's not productive to have slow solutions, and using too many libraries can be a problem. The issue is that Python loops are just too slow, you need to be able to write a loop in C/Cython/etc. Thinking through data structures is also very important.
I get very frustrated that there's this emphasis on parallelism to "solve" speed for Python. Very often the inputs and outputs of the function calls are large enough that you cannot possibly outrace the transfer, pickle and function call overheads, so the more workers you add, the slower it is. Meanwhile if you just write it properly it's 200x faster to start with, and there's no problem.
Re: Python vs Common Lisp, workflow and ecosystem (2019)
#24Earlier quoted context omitted.
> and is fast because the speedy parts aren't in Python. Having worked months with a slew of senior data scientists, this was a bit painful. Python is so slow and those data scientists were very good at coming up with solutions for the issues of the company, but the implementations (using Spacy, Pandas and other libs) had enough Python in them to make them not practical for the company use case. Nice prototypes which…
(I'm the creator of spaCy) If you want to email me at matt@explosion.ai , I'd be interested in the specifics of the algorithm and why the implementation was slow. The idea for something like that keyword extraction algorithm would be that if the Python API is slow, you should just use Cython. The Cython API of spaCy is really fast because the `Doc` is just a `TokenC*`, and the tokens just hold a pointer to their lexe…
Sorry if you think I blamed spaCy for anything; it was not intended; I know it was due to the way Python was used which I tried to convey. Your product is excellent and yes, I probably should've reached out more anyway; I just know how to solve things my way and did not wanted to waste more time (there was an investor deadline).
Cython part sounds good; I will try it out and email you if I get totally stuck, thanks!
Re: Python vs Common Lisp, workflow and ecosystem (2019)
#25Earlier quoted context omitted.
Very much actively doing so on my end where nearly all work in the industry is in Python, with some Matlab, C, C++, and Julia sprinkled in. Python is a great high level language for basically everything, but hardcore low latency apps. I can parse text, connect to databases, do sparse matrix computations on massive matrices, calculate network flows, generate large node-graph diagrams, use a Python based API to connect…
> do sparse matrix computations on massive matrices This is completely impossible to do in the Python language, unless you resort to external tooling written in C or Fortran. Sure, you can call these codes from Python, as you can call them from any other language.
The ecosystem matters. I'm a developer and not a scientist, but having everything inside an environment that's at least workable is a huge boon.
Of course you could call the same functions from the ffi of any other language, but nobody does that for the same reason that nobody writes web applications in C.
I hate python, as far as I'm concerned it's a nightmare hell of a language that does everything wrong, and yet it's probably the language I use the most due to its sheer convenience and massive ecosystem.
Re: Python vs Common Lisp, workflow and ecosystem (2019)
#26Speaking of Common Lisp - the European Lisp Symposium starts tomorrow (May 3 and May 4, https://european-lisp-symposium.org/2021/index.html ). The entire conference will be broadcast on Twitch. Python programmers are invited, too :)
This is exciting. Thanks for the reference.
Re: Python vs Common Lisp, workflow and ecosystem (2019)
#27Earlier quoted context omitted.
(I'm the creator of spaCy) If you want to email me at matt@explosion.ai , I'd be interested in the specifics of the algorithm and why the implementation was slow. The idea for something like that keyword extraction algorithm would be that if the Python API is slow, you should just use Cython. The Cython API of spaCy is really fast because the `Doc` is just a `TokenC*`, and the tokens just hold a pointer to their lexe…
Sorry if you think I blamed spaCy for anything; it was not intended; I know it was due to the way Python was used which I tried to convey. Your product is excellent and yes, I probably should've reached out more anyway; I just know how to solve things my way and did not wanted to waste more time (there was an investor deadline). Cython part sounds good; I will try it out and email you if I get totally stuck, thanks!
Oh, I didn't take it as a pointed criticism or anything. It just seemed like it would be an instructive example.
The underlying point I often make to people is that Python's slowness introduces a lot of incidental complexity, and you find yourself fiddling with numpy or something instead of just writing normal code and expecting it to perform normally.
Re: Python vs Common Lisp, workflow and ecosystem (2019)
#28Earlier quoted context omitted.
Yeah, they're very much doing it. Pandas is huge, libraries like Spacy, NetworkX, etc exist. It's a massive and good ecosystem. Python is the goto for scientific computing in most of the sciences for newer students I'd hazard a guess over the older R and Julia. This will be blindingly obvious if you work in that area. Yes, you can do it in another language, but you're missing out on a lot of stuff that is just done a…
> and is fast because the speedy parts aren't in Python. Having worked months with a slew of senior data scientists, this was a bit painful. Python is so slow and those data scientists were very good at coming up with solutions for the issues of the company, but the implementations (using Spacy, Pandas and other libs) had enough Python in them to make them not practical for the company use case. Nice prototypes which…
Nice prototypes which I then had to fix them or even rewrite to C/C++
Even as someone who 'knows' C and C++ I still find it faster and easier overall to do the exploratory and 'science' part in Python, making sure it works and gives me the answers I need in the format I want etc. only to then rewrite and optimize the slow parts in C or C++ if necessary.
Re: Python vs Common Lisp, workflow and ecosystem (2019)
#29Earlier quoted context omitted.
Yeah, they're very much doing it. Pandas is huge, libraries like Spacy, NetworkX, etc exist. It's a massive and good ecosystem. Python is the goto for scientific computing in most of the sciences for newer students I'd hazard a guess over the older R and Julia. This will be blindingly obvious if you work in that area. Yes, you can do it in another language, but you're missing out on a lot of stuff that is just done a…
> and is fast because the speedy parts aren't in Python. Having worked months with a slew of senior data scientists, this was a bit painful. Python is so slow and those data scientists were very good at coming up with solutions for the issues of the company, but the implementations (using Spacy, Pandas and other libs) had enough Python in them to make them not practical for the company use case. Nice prototypes which…
I've worked in this industry for a bit and had the opposite experience. Most of the slow solutions have been in other languages due to poor design and algo choice. There have been several projects I've been able to rewrite from R to Python where the runtime on a workstation went from days with R to seconds minutes with Python in a tiny tiny VM (like $10 DO box).
Sure maybe a 3 minute task in python that reconciles a few million transactions and builds some very useful projections is too slow for some pipelines, but it worked for my clients.
Re: Python vs Common Lisp, workflow and ecosystem (2019)
#30I like Lisp, and I'm not a fan of e.g. Python's whitespace sensitivity. That said, for niches such as ML and data science, I find you just can't beat the Python ecosystem.
Are people actually doing science with Python or are they talking about doing science? There's so much buggy low quality stuff in that space that I'd write a serious application in C or C++ from scratch. It would be a custom application, sure, but not everything needs to be general. Also, I find Lisp much more natural for mathematical reasoning.
Yes. Python is pretty much the main tool in biology, for example. C or C++ would be abysmal for similar exploratory scientific takes.