Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test for lists in Cython

github.com

Post reply on HN
Print view View original

91–100 of 147 posts

Re: Test for lists in Cython

#91

Post by clircle »

ekianjo wrote: post #15
ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

> Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Same thing in the R ecosystem, it's possible to use multiple cores with the parallel package but there are caveats too.

I do as much data work as possible in data.table, which parallelizes things under the hood, and doesn't seem to require me to configure anything.

Re: Test for lists in Cython

#92

Post by meepmorp »

mhh__ wrote: post #33
ZuLuuuuuu wrote: post #28

Earlier quoted context omitted.

The thing is that the current ecosystem (numpy + scipy + pandas + PyCharm) fits our company structure perfectly (very few software engineers, mostly test engineers who are not very proficient coders). And we already have tens of thousands lines of code. So changing the whole ecosystem just because 5% of the problems are slow is too big of a jump for us. For now, it is easier for us to write a bit of C code for places…

Do you know what you are actually stalled on? i.e. Memory or Instructions, if it's the latter you can probably eek out some more performance just by fiddling with compiler flags (you'd be surprised simultaneously how clever and how utterly braindead some compilers are if you play with them on Compiler Explorer for a bit) Also, if you are running on Intel you might have some luck with Intel's profiling tools as, altho…

> Compiler Explorer

Never seen this before. Thanks!

Re: Test for lists in Cython

#93

Post by mhh__ »

meepmorp wrote: post #92
mhh__ wrote: post #33

Earlier quoted context omitted.

Do you know what you are actually stalled on? i.e. Memory or Instructions, if it's the latter you can probably eek out some more performance just by fiddling with compiler flags (you'd be surprised simultaneously how clever and how utterly braindead some compilers are if you play with them on Compiler Explorer for a bit) Also, if you are running on Intel you might have some luck with Intel's profiling tools as, altho…

> Compiler Explorer Never seen this before. Thanks!

https://xkcd.com/1053/

Enjoy

Re: Test for lists in Cython

#94

Post by O_H_E »

jakobnissen wrote: post #45

I'm a massive Julia fanboy, but I would not extend Python with Julia if I could choose not to. Julia has a _massive_ runtime with a hello-world script consuming 150 MB of RAM, not to mention the dreaded startup-time. It's better and easier to use Julia as your main top-level "glue" language and call Python/Rust/C from Julia. Julia is in many ways a better glue language than Python - better multithreading, easier call…

> _massive_ runtime ... 150MB of RAM, not to mention the dreaded startup-time.

I understand how can this be inconvenient for me and you while scripting. But this is absolutely no problem for julia's currently main target market who runs multi-GB simulations.

Also it is not like they will be spawning a julia instance in a hot loop (THAT would be horrible). You write some julia code, and import it using smth like pyjulia at the beginning of the file.

Re: Test for lists in Cython

#95

Post by [deleted] »

jakobnissen wrote: post #45

I'm a massive Julia fanboy, but I would not extend Python with Julia if I could choose not to. Julia has a _massive_ runtime with a hello-world script consuming 150 MB of RAM, not to mention the dreaded startup-time. It's better and easier to use Julia as your main top-level "glue" language and call Python/Rust/C from Julia. Julia is in many ways a better glue language than Python - better multithreading, easier call…

[deleted]

Re: Test for lists in Cython

#96

Post by diarrhea »

leephillips wrote: post #87
deetsb wrote: post #77

Earlier quoted context omitted.

As I’m clearly missing the bigger picture, what do you find to be super offensive about this? I read the first as the concat symbol applied to an iterable leads to string concated by the concat symbol. I read the second as an iterable broken by split symbol leads to an iterable of the chunks.

It’s hard to remember what order to put things in, because the two methods follow opposite conventions. And that’s because this type of OO design has no obvious method of organization. In the first case, I want to do something to a list: join it into a string. So, clearly, I need a list method? But no, I need to engage in some form of indirection; for some reason, I need to reach for a string method. Even if there is…

The Python FAQ itself adresses this: https://docs.python.org/3/faq/design.html#why-is-join-a-stri...

You'll have to decide if you find that convincing. I understand your point and kind of wish I hadn't read this thread because I'm more torn than before.

Re: Test for lists in Cython

#98

Post by leephillips »

diarrhea wrote: post #96
leephillips wrote: post #87

Earlier quoted context omitted.

It’s hard to remember what order to put things in, because the two methods follow opposite conventions. And that’s because this type of OO design has no obvious method of organization. In the first case, I want to do something to a list: join it into a string. So, clearly, I need a list method? But no, I need to engage in some form of indirection; for some reason, I need to reach for a string method. Even if there is…

The Python FAQ itself adresses this: https://docs.python.org/3/faq/design.html#why-is-join-a-stri... You'll have to decide if you find that convincing. I understand your point and kind of wish I hadn't read this thread because I'm more torn than before.

Not convincing exactly, but I’m glad to know the thinking behind it, thank you. How are you torn?

Re: Test for lists in Cython

#99

Post by chalst »

zozbot234 wrote: post #68
bjourne wrote: post #64

Earlier quoted context omitted.

> As a more general comment, using a GC language as the FFI target from a GC language is begging for difficult-if-not-impossible-to-debug crashes down the line. Not true! What you do is that you keep a registry for objects passed from the host vm to the foreign vm in which you register objects thus transferred. And you use a similar mechanism for objects passed from the foreign vm to the host vm. In CPython, you simp…

> This ... is how many gc:ed runtimes interact with other gc:ed runtimes. It's not that easy. You'd need to register the object as a GC root as long as it's being managed by the foreign GC, and similarly ensure that objects in the foreign VM are properly "de-registered" when the local GC finalizes them. It's far from trivial, particularly when compared with other memory-management strategies.

This is a solved problem: look at how the Lua FFI does it. Lua both is capable of exporting its own garbage collector to its counterparty and has a plug-in system for GC so that Lua can use the counterparty's GC. That allows you to either use Lua's GC for foreign objects or have the counterparty's GC manage Lua's objects.

It's funny the hostility to JIT that some fans of compiled languages have.

Re: Test for lists in Cython

#100

Post by whateveracct »

urschrei wrote: post #9

Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct. https://github.com/urschrei/simplification https://github.com/urschrei/lonlat_bng https://github.com/urschrei/pypolyline Each of those repos has links to the corresponding Rust “shim” libraries that pro…

> As a more general comment, using a GC language as the FFI target from a GC language is begging for difficult-if-not-impossible-to-debug crashes down the line.

Safe interop between two GC'd languages (Haskell and Java) was one motivation of Haskell's new -XLinearTypes extension

https://www.tweag.io/blog/2020-02-06-safe-inline-java/

Generally, the new linear types are exciting but are nascent at the moment. But one extension to them (linear constraint [1]) seems to allow embed the equivalent of Rust's ownership in Hsakell using more primitive features in the type system (to my understanding..)

[1] https://arxiv.org/pdf/2103.06127.pdf

Post reply on HN