Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's up, Python? The GIL removed, a new compiler, optparse deprecated

bitecode.dev

Post reply on HN
Print view View original

161–170 of 302 posts

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#161

Post by HdS84 »

Turskarama wrote: post #143
OkGoDoIt wrote: post #133

Earlier quoted context omitted.

Whatever happened to ironpython? I used to do a lot of C# development and remember dabbling with ironpython back in the day. It seemed like it was important to Microsoft, .Net added the whole concept of dynamic data types mostly to support ironpython and ironruby. But I never really used python much until recently, so of course when I finally needed to do python I looked for ironpython and it doesn’t appear to be a t…

It is still a thing, but it's open source now instead of maintained by Microsoft. There was a release that finally supports Python 3 in December last year. I don't know how useful it is really, if you really want performance then you probably shouldn't choose Python to begin with, or you use the libraries which may not be compatible with IronPython. These days it barely takes me longer to build a simple script in C#…

It's so so. Pythons core value is it's huge stack of lib's. And most important fall down with IP due to them using c and so on.

When we needed python c# interop it was better to use python.net and integrate that way. Annoying to setup but when it works you can get both to work seamlessly

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#162

Post by tsimionescu »

jgalt212 wrote: post #97
nwallin wrote: post #53

Earlier quoted context omitted.

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

> Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. I assume mod_wsgi under apache was not the answer here due to memory constraints. That being said, why not serve from disk and use redis for a cache. This should work well unless the queries had high cardinality.

Serve what from disk? If they are using python, they are almost certainly writing am api server, not static files.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#163

Post by tsimionescu »

mike_ivanov wrote: post #75
__d wrote: post #67

Earlier quoted context omitted.

Python is both an interpreter, and quite dynamic. Both of these lead to lower performance when compared to less dynamic, compiled solutions. All of Java, Go, and .NET are compiled and (much) less dynamic. This is absolutely an expected outcome.

"absolutely an expected outcome." Good day. Is it the right time to talk to you about Common Lisp?

To be fair, if you use CL in a similarly dynamic way as Python (don't compile anything, don't add any declarations etc) it won't be that much faster. You'll get some boost out of the stdlib stuff being compiled already, but otherwise it will incur similar performance penalties.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#164

Post by amrx101 »

nwallin wrote: post #53
wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

I dont really partake in programming "wars", but the idea of launching a set of separate processes instead of separate threads to do a bunch of IOs has always seem to be weird to me. Yes, I have built software using Python. Yes, I have done things as you suggest. Now I use asyncio, since the syntax has matured and I finally understand coroutines, runners, tasks etc. Lets see where the GIL less Python takes us.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#165

Post by tsimionescu »

qbasic_forever wrote: post #70
wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

Are you just reading from this data structure? If so I wouldn't do any locking or threading, I'd just use asyncio to serve up read requests to the data and it should scale quite well. Multithreading/processing is best for CPU limited workloads but this sounds like you're really just IO-bound (limited by the very high IO of reading from that data structure in memory). If you're allowing writes to the shared data struc…

Reading from memory is really not IO. Perhaps you're suggesting doing something like mmapping a file to memory, putting the data structure in that memory, and then using asyncio on the file to serve things, but this would only work if you can compute byte ranges inside the file to serve ahead of time, in which case there are much simpler solutions anyway. Most likely, when receiving a query they need to actually search through the datastructure based on the query, and it's very likely that this is the bottleneck, not just reading some memory.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#166

Post by tsimionescu »

School-Cotton wrote: post #92
ctoth wrote: post #72

Earlier quoted context omitted.

I notice you used the strong emotional word "ruining" when talking about the effect on Python of this change. Why do you believe an obscure runtime concurrency detail which will make more things possible will "ruin" the language? Now match and :=? Those definitely ruin the language. ;-) But seriously, relax, nothing bad is happening here. It's not just people who have to use the torch launcher who have been bitten by…

> Why do you believe an obscure runtime concurrency detail It is not obscure. It will make it much more difficult to write native-code extensions which is IMO the whole point of Python.

The point of Python in your opinion is to write non-Python code?

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#167

Post by SanderNL »

wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

It sounds I/O heavy, but you mention it being CPU-heavy in which case I’d say Python is just not the right tool for the job although you may be able to cope with multiprocessing.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#168

Post by Nursie »

qbasic_forever wrote: post #73
andrewstuart wrote: post #3

From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?

Correct, it will help with CPU-limited, embarrassingly parallelizable problems... which are much less common than you think.

You don’t need embarrassingly parallel problems, you just need code doing lots of the same thing at the same time for this to be a win.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#169

Post by BiteCode_dev »

wodenokoto wrote: post #140

When it was an in dev project, I felt the consensus on HN was that it was amazing work and a shame that it looked like the steering committee wouldn’t adopt it. Now they have and everyone seems to hate it.

It's the eternal pendulum:

- take no risk, and people will blame the project for being static.

- take risks, and people will blame the project for being reckless.

E.G:

- don't adopt a new feature, and your language is old, becoming irrelevant, and a wave of comments will tell you how they just can't use it for X because they don't have it.

- break compat, and you will have a horde stating you don't care about users that need stability. You got one comment in this thread talking about "the python treadmill"!

And all that for an open source project most don't contribute to and never paid a dime for.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#170

Post by felipetrz »

coldtea wrote: post #86
nwallin wrote: post #53

Earlier quoted context omitted.

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

> Use the python multiprocessing module. If you've already written it with the multithreading module, it is a drop in replacement. Your data structure will live in shared memory Only if it can be immutable. So it can't be shared and changed by multiple processes as needed (with synchronization). And even if you can have it mostly immutable, if you need to refresh it (e.g. after some time read a newer large file from…

For this use case it would be better to put the data in a shared SQLite database than relying on multiprocessing CoW.

Even accessing objects from the shared memory would cause the reference counter to increment and the data would be copied, causing a memory usage explosion.

Post reply on HN