Is there a Nobel category for “major Python improvements”?
A viable solution for Python concurrency
191–200 of 366 posts
Re: A viable solution for Python concurrency
#192If the Python maintainers doesn’t want to approve this, Gross should talk to the Pypi developers.
> If the Python maintainers doesn’t want to approve this, Gross should talk to the Pypi developers.
I suspect you mean “PyPy” which is a very different thing than “pypi”.
Re: A viable solution for Python concurrency
#193Earlier quoted context omitted.
You are likely being downvoted because most claims about the pain of a Python 3 transition are inflated/hyperbole. It took less than a day to migrate all my code to Python 3. And by "less than a day" I mean "less than 2 hours". Granted, bigger projects would take longer, but saying stuff like "10+ years of pain" is ridiculous. Probably less than 1% of projects had serious issues with the migration. We just hear of a…
The entire Python community was in pain over Python 3 for 10 years, even if migrating any particular program wasn't much trouble. If you want to contest the notion that there was pain, then fine: most of the community simply ignored Python 3 for 10 years, because there was no reason until quite late in the process to worry about it. I myself never bothered migrating any of my Python 2 stuff. It might not be difficult…
Some people don’t get that migrating large legacy code bases from 2 to 3 is not trivial. Your dropbox example is a good one: the migration took 3 years to complete. What else could dropbox have achieved with a 3 year long project if they didn’t have to migrate from 2 to 3?
Re: A viable solution for Python concurrency
#194This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Elixir is a great example.
Because we started with python for historical reasons when the company was small and performance didn’t matter, and now we’re stuck with it
Re: A viable solution for Python concurrency
#195I feel like Gvr just doesnt want to change things, Feels doomed This has been a problem for like 20 years and they have refused fixes before. And there have been fixes. They just don't see this as important it's practically a religion that its a thing they wont change
I disagree entirely. The last few releases of Python have made significant changes to the language, coinciding with the project becoming community-led after Guido stepped down.
> The last few releases of Python have made significant changes to the language, coinciding with the project becoming community-led after Guido stepped down.
A lot of that is stuff that is enabled by, or was blocked pending, the new parser; I don't think it was blocked on Guido, and Guido haa hardly stopped being active and influential since stepping down as BDfL
Re: A viable solution for Python concurrency
#196Free Dask cluster available to use at tcp://3.216.44.221:8786
tldr: you can use our 4-computer Dask cluster (including two GPUs, a GTX 1080Ti and a Radeon 6900XT) at no cost. If you need even more computing power, message me (george AT lindenhoney DOT com).
FAQ
What is Dask? Dask is a Python framework for distributed computing, designed to enable data scientists to process large amounts of data and huge computations on a scale from 1 to thousands of computers.
How do I use the cluster you created? Install Dask (python -m pip install "dask[complete]" from dask.distributed import Client client = Client(tcp://3.216.44.221:8786) Use it! https://examples.dask.org
Do I need to sign up? No. You can literally just use it as above from any Python script or notebook.
Why are you making it available? I’m looking to better understand how people use Dask and how to best make it available to data scientists as an easy to use service.
What can I use the cluster for? Anything you want such as data science, web crawling (except spam, porn, illegal activities or DoS) etc. Please be nice.
Can I hack your cluster? Probably not. But ff you find any security issues please let us know instead and we will fix them.
What’s the catch? Do you track my usage? None. These are my gaming/closet computers that I just want people to use. I do track anonymous function calls people make and the total volume of data that passes through the system (bandwidth, disk usage, RAM usage etc) but I do not look at or save any of your data.
How can I add my own worker machines to the cluster? Easy - set up Dask worker(s) and point them to tcp://3.216.44.221:8786
How about security and confidentiality? These are computers I have at home so all you have is my promise that I won’t look/save/copy your data or code. Pinky swear. However, it is possible for other unfriendly users to look at your data if you save it, even temporarily, on the disk, since all Dask tasks run under the same user.
Who are you? https://www.linkedin.com/in/georgeciobanunyc
What if someone else who uses the cluster messes up with my data, copies it or otherwise takes it down? Use your best judgement. I hardened the cluster’s security to the best of my ability but I can’t guarantee that someone else won’t mess with them. If you work for the NSA/a bank/IRS/etc you should definitely not upload sensitive data to these computers.
I need more GPUs or a bigger (hundreds or thousands of computers) cluster! Glad to help, just email me (see above).
I need a larger cluster but private (using AWS, Azure, GCP) Same as above, email me and I can help.
What are the computer specs? Two of them are AMD processors with 8 physical cores each, the other is a quad core Intel and the last one is a Mac (Intel, 4 cores). The dedicated graphics cards (GPUs) are on the AMD computers.
Re: A viable solution for Python concurrency
#197Earlier quoted context omitted.
The entire Python community was in pain over Python 3 for 10 years, even if migrating any particular program wasn't much trouble. If you want to contest the notion that there was pain, then fine: most of the community simply ignored Python 3 for 10 years, because there was no reason until quite late in the process to worry about it. I myself never bothered migrating any of my Python 2 stuff. It might not be difficult…
IMO, it's a good thing that decode() doesn't on strings in Python 3.
There is a case to be made either way. The point is that it's yet another quirky difference that pops up sometimes, and using python3 keeps bringing up a stream of such issues.
It does seem in the spirit of Python's duck typing to be able to say '616263'.decode('hex') and get 'abc', and that does work fine in py2. Try it in py3 and you get a type error. So ok, convert the string to bytes, e.g. by saying b'616263' instead of '616263'. That doesn't work either, but I'll leave the experiment to you.
Re: A viable solution for Python concurrency
#198Many of the additions to Python in the past decade have been very impressive, but am I wrong in thinking that they suffer from a kind of diminishing marginal benefit? If I am building a project where concurrency or asynchrony are essential, am I going to choose Python? If I need to bolt these on to an existing project to meet a deadline, how much runway do I really get from these enhancements before I hit the limitat…
> am I going to choose Python?
IMO most of the additions to Python in the past decade haven’t been aimed at people choosing a language, but at people working on large existing Python codebases who have too much inertia to change languages.
The cause of this is that the people in charge of language design (first Guido at Dropbox and now members of the Steering Council) are in exactly that situation.
The effect is that Python has lost focus on what it was originally good for (executable pseudocode for scripting) in favour of adding mediocre support for concurrency, static typing, and now pattern matching.
Re: A viable solution for Python concurrency
#199Earlier quoted context omitted.
The entire Python community was in pain over Python 3 for 10 years, even if migrating any particular program wasn't much trouble. If you want to contest the notion that there was pain, then fine: most of the community simply ignored Python 3 for 10 years, because there was no reason until quite late in the process to worry about it. I myself never bothered migrating any of my Python 2 stuff. It might not be difficult…
Some people don’t get that migrating large legacy code bases from 2 to 3 is not trivial. Your dropbox example is a good one: the migration took 3 years to complete. What else could dropbox have achieved with a 3 year long project if they didn’t have to migrate from 2 to 3?
And Guido van Rossum himself was working at dropbox at the time, contributing to that port. Imagine how much harder it might have been without him.
Re: A viable solution for Python concurrency
#200I might have a controversial or unpopular opinion - I do not think python should try to be concurrent or any more performant than it already is.
Python is the tool I pick for quick scripting, not for highly performant systems - there are languages and run times for that.
Sometime highly productive software does not need to be highly performant, and that does not make the software any more or less valuable. And sometimes speed of iteration is more valuable than speed of run time execution.