Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test for lists in Cython

github.com

Post reply on HN
Print view View original

121–130 of 147 posts

Re: Test for lists in Cython

#121

Post by jhgb »

leephillips wrote: post #117
jhgb wrote: post #115

Earlier quoted context omitted.

> Classes bind together data and methods Not in CLOS, from which Julia's mechanisms are derived (which leads to the question why not CLOS but Python of all things should be used as a source of "normal sense" of anything Julia-related). In CLOS, classes bind together data, generic functions name abstract operations, and methods represent specific code that deals with implementing a generic function for a particular co…

Well, it seems as if we were just at cross purposes due to terminology. Although multiple dispatch in CL (and Perl) predates Julia, I was not aware that Julia’s design derived from it. Do you have a reference that traces this?

I find it virtually certain that it does derive from it. Considering that one of the authors of Julia wrote Julia's front-end in Lisp (https://github.com/JuliaLang/julia/blob/master/src/julia-par... and some other files in the same directory), it would have been astonishing for CLOS to not have major impact on the design. There's also some relevant statements in a paper on Julia's design (https://dl.acm.org/doi/10.1145/3276490) in the part on multiple dispatch in section 7. Related Work, where CLOS and its "algebraic cousin" Dylan are mentioned. I got the impression that Julia's object system is basically CLOS without quite a few of CLOS' complexities such as inheritance (which in CLOS necessitates some advanced extension facilities to cover some corner cases if method lookup doesn't do what you want it to do if you're attempting a highly complex application model). The nice effect of those feature removals was that in many cases monomorphization of call sites in emitted native code is possible, which is presumably the other reason for those feature removals: suddenly even primitive operations such as +, * etc. can be generics without incurring (most of the time) dispatch cost at runtime. That (primitive operations being generic functions) is not the case in CLOS, although that can also very well be attributed to backwards compatibility efforts in Common Lisp.

Interestingly enough, in Julia's documentation, the section "Noteworthy Differences from other Languages" (https://docs.julialang.org/en/v1/manual/noteworthy-differenc...) compares Julia to only several relevant languages, which are: Matlab, R, Python, C/C++, and...Common Lisp, of all things. I very strongly doubt that this is a coincidence.

Re: Test for lists in Cython

#122

Post by bennofs »

bjourne wrote: post #64
urschrei wrote: post #9

Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct. https://github.com/urschrei/simplification https://github.com/urschrei/lonlat_bng https://github.com/urschrei/pypolyline Each of those repos has links to the corresponding Rust “shim” libraries that pro…

> As a more general comment, using a GC language as the FFI target from a GC language is begging for difficult-if-not-impossible-to-debug crashes down the line. Not true! What you do is that you keep a registry for objects passed from the host vm to the foreign vm in which you register objects thus transferred. And you use a similar mechanism for objects passed from the foreign vm to the host vm. In CPython, you simp…

But it breaks down if you have cross-language cycles, because no garbage collector of either language will see the full cycle.

Re: Test for lists in Cython

#123

Post by [deleted] »

Gravityloss wrote: post #75
leephillips wrote: post #63

Earlier quoted context omitted.

The standard way to make a histogram in Julia is histogram(data) Using the latest version (1.6 - although 1.6.1 just came out) the time to first plot is just a few seconds. After that, plotting in the REPL is instantaneous. I probably don’t understand what you’re getting at when you speak of making frequent changes to code. REPL-based development in Julia is excellent, and there are Pluto notebooks as well.

The way I would like to work is to have the repl open on the left hand side of the screen and code editor (like Sublime text, I'm sure many use Vim) on the right hand side. I would run the code in repl (just using up-arrow and enter), get some plot, modify the code in the editor, save it, and rerun it repl. Repl is used for connecting inputs to the program, not for editing. Often times you want to develop some small…

[deleted]

Re: Test for lists in Cython

#124

Post by ZuLuuuuuu »

joshuaellinger wrote: post #72
cfcf14 wrote: post #51

Earlier quoted context omitted.

Have you been successful in implementing non-trivial computational code in numba/numpy? I've always found it starts to really break for anything which isn't really trivial, and the errors are mostly non-prescriptive and highly verbose.

I just implemented both a CSV parser and an address standardizer in numba (both CPU and GPU) running in parallel feed through a message queue with a bunch of workers subprocs. It takes a bit of getting used to but the performance gains on impressive. Basically, my bottlenecks shift from compute to i/o. I think you have to balance it against writing in C/C++. Mentally, it is basically the same work as writing in C (yo…

> I think the unappreciated advantage of python is that I have to abandon all pretense of caring about speed and just get stuff working. It basically solves the pre-mature optimization problem for me

I feel the same way. With Python I just write the simplest algorithm that first comes to my mind, even though I know that it is not the most optimized way of doing things. But most of the time I am surprised that it works so fast that I realize I actually don't need to optimize it.

And being able to create and easily manipulate dictionaries and tuples also allows me to create efficient data structures very quickly.

Re: Test for lists in Cython

#125

Post by dr_zoidberg »

The cython code is a bit messy. Changing from:

    cpdef float iterate_list(a_list):

        cdef double count = 0
        cdef int i, j
        for i in range(len(a_list)):
            internal_list = a_list[i]
            for j in range(len(internal_list)):
                count += internal_list[j]
        print(count)
        return count
To:

    cpdef float iterate_list(list a_list):

        cdef double count = 0
        cdef double val = 0
        cdef list ilist
        for ilist in a_list:
            for val in ilist:
                count += val
        print(count)
        return count
Speeds up the iterate_list function an order of magnitude. On my PC:

    In [9]: %timeit list_cy.iterate_list(a_list)
    1000000.0007792843
    1000000.0007792843
    1000000.0007792843
    1000000.0007792843
    1000000.0007792843
    1000000.0007792843
    1000000.0007792843
    1000000.0007792843
    385 ms ± 6.15 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

    In [10]: %timeit list_cyo.iterate_list(a_list)
    1000000.0007792843
    1000000.0007792843
    1000000.0007792843
    1000000.0007792843
    1000000.0007792843
    1000000.0007792843
    1000000.0007792843
    1000000.0007792843
    2.71 s ± 182 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
(yeah, I kept the prints that the code has)

Where list_cy is the fixed code and list_cyo is the original code. Even then, iterating over a list of lists is _definitely not_ the optimal way you'd face a problem of this kind. Numpy arrays and memoryviews would be the correct tool to use.

Re: Test for lists in Cython

#126

Post by optimalsolver »

dr_zoidberg wrote: post #125

The cython code is a bit messy. Changing from: cpdef float iterate_list(a_list): cdef double count = 0 cdef int i, j for i in range(len(a_list)): internal_list = a_list[i] for j in range(len(internal_list)): count += internal_list[j] print(count) return count To: cpdef float iterate_list(list a_list): cdef double count = 0 cdef double val = 0 cdef list ilist for ilist in a_list: for val in ilist: count += val print(c…

Thanks for this.

Will Julia advocates every use honest benchmarks to make their language look good? I doubt it.

Re: Test for lists in Cython

#127

Post by leephillips »

jhgb wrote: post #121
leephillips wrote: post #117

Earlier quoted context omitted.

Well, it seems as if we were just at cross purposes due to terminology. Although multiple dispatch in CL (and Perl) predates Julia, I was not aware that Julia’s design derived from it. Do you have a reference that traces this?

I find it virtually certain that it does derive from it. Considering that one of the authors of Julia wrote Julia's front-end in Lisp ( https://github.com/JuliaLang/julia/blob/master/src/julia-par... and some other files in the same directory), it would have been astonishing for CLOS to not have major impact on the design. There's also some relevant statements in a paper on Julia's design ( https://dl.acm.org/doi/10.…

Thank you for that interesting reply.

Re: Test for lists in Cython

#128

Post by dr_zoidberg »

optimalsolver wrote: post #126
dr_zoidberg wrote: post #125

The cython code is a bit messy. Changing from: cpdef float iterate_list(a_list): cdef double count = 0 cdef int i, j for i in range(len(a_list)): internal_list = a_list[i] for j in range(len(internal_list)): count += internal_list[j] print(count) return count To: cpdef float iterate_list(list a_list): cdef double count = 0 cdef double val = 0 cdef list ilist for ilist in a_list: for val in ilist: count += val print(c…

Thanks for this. Will Julia advocates every use honest benchmarks to make their language look good? I doubt it.

OP has an open pull request on his repo[0] where someone made basically this same change (different names for the variables, but same idea).

According to OP, he tried this but it resulted in slower execution for him. I'm not sure he really followed what that PR says. In the case of the submitter, it says on their machine it gave a 2x speedup (smaller than my ~7x, but still significant).

[0] https://github.com/00sapo/cython_list_test/pull/3

Re: Test for lists in Cython

#129

Post by anotheranonym »

optimalsolver wrote: post #126
dr_zoidberg wrote: post #125

The cython code is a bit messy. Changing from: cpdef float iterate_list(a_list): cdef double count = 0 cdef int i, j for i in range(len(a_list)): internal_list = a_list[i] for j in range(len(internal_list)): count += internal_list[j] print(count) return count To: cpdef float iterate_list(list a_list): cdef double count = 0 cdef double val = 0 cdef list ilist for ilist in a_list: for val in ilist: count += val print(c…

Thanks for this. Will Julia advocates every use honest benchmarks to make their language look good? I doubt it.

1000% this. Multiple times I’ve encountered a Julia benchmark claiming to show its superiority in a task I routinely perform. And every time the pro Julia benchmark turned out to be total BS.
Post reply on HN