Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Free-threaded CPython is ready to experiment with

labs.quansight.org

Post reply on HN
Print view View original

301–310 of 398 posts

Re: Free-threaded CPython is ready to experiment with

#301

Post by GTP »

mixmastamyk wrote: post #174
GTP wrote: post #46

Earlier quoted context omitted.

Then we have very different ideas of what proper typing is :D Look at this function, can you tell me what it does? def plus(x, y): return x+y If your answer is among the lines of "It returns the sum x and y" then I would ask you who said that x and y are numbers. If these are strings, it concatenates them. If instead you pass a string and a number, you will get a runtime exception. So not only you can't tell what a f…

> you can't tell what a function does just by looking at it You just did tell us what it does by looking at it, for the 90% case at least. It might be useful to throw two lists in there as well. Throw a custom object in there? It will work if you planned ahead with dunder add and radd. If not fix, implement, or roll back.

> You just did tell us what it does by looking at it, for the 90% case at least

The problem is that you can't know if the function is going to do what you want it to do without also looking at the context in which it is used. And what you pass as input could be dependent on external factors that you don't control. So I prefer the languages that let me know what happens in 100% of the cases.

Re: Free-threaded CPython is ready to experiment with

#302

Post by ignoramous »

bausgwi678 wrote: post #249
pizza234 wrote: post #231

Earlier quoted context omitted.

> using simple threads instead of dealing with the massive overhead and complexity and bugs of using something like multiprocessing. Depending on the domain, the reality can be the reverse. Multiprocessing in the web serving domain, as in "spawning separate processes", is actually simpler and less bug-prone, because there is considerably less resource sharing. The considerably higher difficulty of writing, testing an…

Using multiple processes is simpler in terms of locks etc, but python libraries like multiprocessing or even subprocess.popen[1] which make using multiple processes seem easy are full of footguns which cause deadlocks due to fork-safe code not being well understood. I’ve seen this lead to code ‘working’ and being merged but then triggering sporadic deadlocks in production after a few weeks. The default for multiproce…

> The default for multiprocessing is still to fork (fortunately changing in 3.14)

If I may: Changing from fork to what?

Re: Free-threaded CPython is ready to experiment with

#303

Post by zo1 »

GTP wrote: post #294
kortex wrote: post #140

Earlier quoted context omitted.

https://github.com/python/mypy > Python itself doesn't do that The type syntax is python. MyPy is part of Python. It's maintained by the python foundation. Mypy is not part of CPython because modularity is good, the same way that ANSI C doesn't compile anything, that's what gcc, clang, etc are for. Mojo is literally exactly the same way, the types are optional, and the tooling handles type checking and compilation.

> Mojo is literally exactly the same way. No, because in Mojo, type checking is part of the language specification: you need no external tool for that. Python defines a syntax that can be used for type checking, but you need an external tool to do that. GCC does type checking because it's defined in the language specification. You would have a situation analogous to Python only if you needed GCC + some other tool for…

You're really splitting hairs here all to prove some sort of "type checking isn't included with python" property. Even if you're technically right, half the world really doesn't care and most code being churned out in Python is type-hinted, type-checked, and de-facto enforced at design time.

It's honestly winning the long-term war because traditional languages have really screwed things up with infinite and contrived language constructs and attempts just to satisfy some "language spec" and "compiler", whilst still trying to be expressive enough for what developers need and want to do safely. Python side-stepped all of that, has the perfect mix of type-checking and amazing "expressibility", and is currently proving that it's the way forward with no stopping it.

Re: Free-threaded CPython is ready to experiment with

#304

Post by GTP »

hot_gril wrote: post #57
GTP wrote: post #46

Earlier quoted context omitted.

Then we have very different ideas of what proper typing is :D Look at this function, can you tell me what it does? def plus(x, y): return x+y If your answer is among the lines of "It returns the sum x and y" then I would ask you who said that x and y are numbers. If these are strings, it concatenates them. If instead you pass a string and a number, you will get a runtime exception. So not only you can't tell what a f…

When is the last time you had a bug IRL caused by passing the wrong kind of thing into plus(x, y), which your tests didn't catch?

It never happened to me, because I don't use Python ;)

On a more serious note, your comment actually hints at an issue: unit testing is less effective without static type checking. Let's assume I would like to sum x and y. I can extensively test the function and see that it indeed correctly sums two numbers. But then I need to call the function somewhere in my code, and whether it will work as intended or not depends on the context in which the function is used. Sometimes the input you pass to a function depends from some external source outside your control, an if that's the case you have to resort to manual type checking. Or use a properly typed language.

Re: Free-threaded CPython is ready to experiment with

#305

Post by zo1 »

zitterbewegung wrote: post #132
nine_k wrote: post #21

Earlier quoted context omitted.

A language server in your IDE kicks in much earlier, and is even more helpful.

I haven't used an IDE that has that but it is still just giving you a hint that there is an error and the interpreter is not throwing an error which was my point.

> "I haven't used an IDE that has that but it is still just giving you a hint that there is an error and the interpreter is not throwing an error which was my point."

At this point, I'm not sure how one is to take your opinion on this matter. Just like me coding some C# or Java in notepad and then opining to a Java developer audience about the state of their language and ecosystem.

Re: Free-threaded CPython is ready to experiment with

#306

Post by GTP »

zo1 wrote: post #303
GTP wrote: post #294

Earlier quoted context omitted.

> Mojo is literally exactly the same way. No, because in Mojo, type checking is part of the language specification: you need no external tool for that. Python defines a syntax that can be used for type checking, but you need an external tool to do that. GCC does type checking because it's defined in the language specification. You would have a situation analogous to Python only if you needed GCC + some other tool for…

You're really splitting hairs here all to prove some sort of "type checking isn't included with python" property. Even if you're technically right, half the world really doesn't care and most code being churned out in Python is type-hinted, type-checked, and de-facto enforced at design time. It's honestly winning the long-term war because traditional languages have really screwed things up with infinite and contrived…

I'm not saying that no one should use Python, I'm just saying why I don't like it. But if you instead like it I will not try to stop you using it.

This said, if most people use type hints and the proper tooling to enforce type checking, I would say this would be a good reason to properly integrate (optional) static typing in the language: it shows that most programmers like static typing. The problem I focused on in my example isn't the only advantage of a type system.

Re: Free-threaded CPython is ready to experiment with

#307

Post by hot_gril »

kstrauser wrote: post #291
hot_gril wrote: post #226

Earlier quoted context omitted.

Maybe a place to look for obvious use cases is in other languages. JS doesn't have threads, but Swift does. The reason I can't think of one is, free threads are most useful for full parallelism that isn't "embarrassingly parallel," otherwise IPC does fine. So far, I've rarely seen that. Best example I deal with was a networking project with lots of communication across threads, and that one was too performance-sensit…

That's the kind of thing I stumble across all the time. Indexing all the symbols in a codebase: results = Counter() for file in here.glob('*.py'): symbols = parse(file) results.update(symbols) Scanning image metadata: for image in here.glob('*.png'): headers = png.Reader(image) ... Now that I think about it, most of my use cases involve doing expensive things to all the files in a directory, but in ways where it'd be…

These are "embarassingly parallel" examples that multiprocessing is ok for, though. There was always the small caveat that you can't pickle a file handle, but it wasn't a real problem. Threads are more useful if you have lots of shared state and mutexes.

I think these examples would also perform well with GIL'd threads, since the actual Python part is just waiting on blocking calls that do the expensive work. But maybe not.

Re: Free-threaded CPython is ready to experiment with

#308

Post by hot_gril »

GTP wrote: post #304
hot_gril wrote: post #57

Earlier quoted context omitted.

When is the last time you had a bug IRL caused by passing the wrong kind of thing into plus(x, y), which your tests didn't catch?

It never happened to me, because I don't use Python ;) On a more serious note, your comment actually hints at an issue: unit testing is less effective without static type checking. Let's assume I would like to sum x and y. I can extensively test the function and see that it indeed correctly sums two numbers. But then I need to call the function somewhere in my code, and whether it will work as intended or not depends…

This isn't an actual problem people encounter in unit testing, partially because you test your outer interfaces first. Also, irl static types often get so big and polymorphic that the context matters just as much.

Re: Free-threaded CPython is ready to experiment with

#309

Post by quietbritishjim »

eigenvalue wrote: post #12

Really excited for this. Once some more time goes by and the most important python libraries update to support no GIL, there is just a tremendous amount of performance that can be automatically unlocked with almost no incremental effort for so many organizations and projects. It's also a good opportunity for new and more actively maintained projects to take market share from older and more established libraries if th…

If you're worried about performance then much of your CPU time is probably spent in a C extension (e.g. numpy, scipy, opencv, etc.). Those all release the GIL so already allow parallelisation in multiple threads. That even includes many functions in the standard library (e.g. sqlite3, zip/unzip). I've used multiple threads in Python for many years and never needed to break into multiprocessing.

But, for sure, nogil will be good for those workloads written in pure Python (though I've personally never been affected by that).

Re: Free-threaded CPython is ready to experiment with

#310

Post by zo1 »

hot_gril wrote: post #84
BugsJustFindMe wrote: post #73

Earlier quoted context omitted.

I feel like "asyncio is the future" was invented by the same people who think it's totally normal to switch to a new javascript web framework every 6 months.

JS had an event loop since the start. It's an old concept that Python seems to have lifted, as did Rust. I used Python for a decade and never really liked the way it did threads.

Python's reactor pattern, or event loop as you call it, started with the "Twisted" framework or library. And that was first published in 2003. That's a full 6 years before Node.js was released which I assume was the first time anything event-loopy started happening in the JS world.

I forgot to mention that it came into prominence in the Python world through the Tornado http server library that did the same thing. Slowly over time, more and more language features were added to give native or first-class-citizen support to what a lot of people were doing behind the scenes (in sometimes very contrived abuses of generator functions).

Post reply on HN