Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geospatial data science with Julia

juliaearth.github.io

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Geospatial data science with Julia

#41

Post by adgjlsfhk1 »

th0ma5 wrote: post #40
adgjlsfhk1 wrote: post #39

Earlier quoted context omitted.

For the same reason that C/C++ allow inline assembly? Languages come in roughly 3 speeds. Slow (e.g. python/R), mostly not slow (e.g. Java/Go), and not slow (e.g. C/Rust). If you want actually fast code (e.g. the speed of BLAS/FFTW etc) you need the combination of a not slow language, code generation, and often hand-coded assembly for the most performance critical parts.

I noticed you didn't mention Julia explicitly this time because when you outline the abstractions like this it seems silly to claim something about Julia magically solves the position and purpose of these layers. I can write a PySpark job based on a tutorial that would run circles around a single core Julia process that was designed with contradictory requirements. I just don't see how Julia gets away with claiming i…

To be explicit about where julia fits in here, Julia is a "not slow" language (you could make an argument for it being on the faster end of "mostly not slow" due to GC) that also has enough high level features (higher order functions, macros, memory management, general ease of use) to work as a high level language. You absolutely can write a distributed python codebase that runs faster than single core julia, but doing so will likely be harder than writing the distributed/threaded Julia code that is way faster than PySpark.

Re: Geospatial data science with Julia

#42

Post by th0ma5 »

adgjlsfhk1 wrote: post #41
th0ma5 wrote: post #40

Earlier quoted context omitted.

I noticed you didn't mention Julia explicitly this time because when you outline the abstractions like this it seems silly to claim something about Julia magically solves the position and purpose of these layers. I can write a PySpark job based on a tutorial that would run circles around a single core Julia process that was designed with contradictory requirements. I just don't see how Julia gets away with claiming i…

To be explicit about where julia fits in here, Julia is a "not slow" language (you could make an argument for it being on the faster end of "mostly not slow" due to GC) that also has enough high level features (higher order functions, macros, memory management, general ease of use) to work as a high level language. You absolutely can write a distributed python codebase that runs faster than single core julia, but doi…

Yeah citation needed on that one, but it is a dumb hypothetical on my part to illustrate the problem. Another hypothetical, how do you get junior people to support your inline ASM? If it makes this easy to do, it makes the technical debt that much more rampant.

Re: Geospatial data science with Julia

#43

Post by adgjlsfhk1 »

th0ma5 wrote: post #42
adgjlsfhk1 wrote: post #41

Earlier quoted context omitted.

To be explicit about where julia fits in here, Julia is a "not slow" language (you could make an argument for it being on the faster end of "mostly not slow" due to GC) that also has enough high level features (higher order functions, macros, memory management, general ease of use) to work as a high level language. You absolutely can write a distributed python codebase that runs faster than single core julia, but doi…

Yeah citation needed on that one, but it is a dumb hypothetical on my part to illustrate the problem. Another hypothetical, how do you get junior people to support your inline ASM? If it makes this easy to do, it makes the technical debt that much more rampant.

You seem really hung up on this inline asm thing. It's not like most julia code is just inline assembly. It's something that at a rough estimate, .5% of packages use to let you squeeze out the final drops of performance that then gets wrapped in an API that looks like normal julia code. This isn't any different from C/C++ which also in some low level code bases will have calls to compiler directives or inline assembly.

Re: Geospatial data science with Julia

#44

Post by th0ma5 »

adgjlsfhk1 wrote: post #43
th0ma5 wrote: post #42

Earlier quoted context omitted.

Yeah citation needed on that one, but it is a dumb hypothetical on my part to illustrate the problem. Another hypothetical, how do you get junior people to support your inline ASM? If it makes this easy to do, it makes the technical debt that much more rampant.

You seem really hung up on this inline asm thing. It's not like most julia code is just inline assembly. It's something that at a rough estimate, .5% of packages use to let you squeeze out the final drops of performance that then gets wrapped in an API that looks like normal julia code. This isn't any different from C/C++ which also in some low level code bases will have calls to compiler directives or inline assembl…

The way it is brought up in arguments about how if what you're doing in Julia is slow you can put in the ASM directly in arguments leaves me with the impression that is nonetheless a core part of the "faster than " claims at least. And that's a cop out.

Re: Geospatial data science with Julia

#45

Post by adgjlsfhk1 »

th0ma5 wrote: post #44
adgjlsfhk1 wrote: post #43

Earlier quoted context omitted.

You seem really hung up on this inline asm thing. It's not like most julia code is just inline assembly. It's something that at a rough estimate, .5% of packages use to let you squeeze out the final drops of performance that then gets wrapped in an API that looks like normal julia code. This isn't any different from C/C++ which also in some low level code bases will have calls to compiler directives or inline assembl…

The way it is brought up in arguments about how if what you're doing in Julia is slow you can put in the ASM directly in arguments leaves me with the impression that is nonetheless a core part of the "faster than " claims at least. And that's a cop out.

If I remember correctly, this came up originally in the context of comparing heavily optimized Julia code to C code that had inline assembly, in which a statement was made that Julia was obviously slower than C because the C code had hand written assembly in it. Julia, like C, sometimes needs hand-coded assembly to achieve maximal speed. 80% of a fast programming language is not having semantics that are fundamentally opposed to speed (i.e. object oriented architectures that require pointer chasing, using arbitrary precision numbers everywhere, or eval semantics that prevent interpreting rather than compiling code). Languages that don't make those kinds of mistakes are "not slow". i.e. if you write similar code in them, you will end up with similar performance to C.

Re: Geospatial data science with Julia

#46

Post by th0ma5 »

adgjlsfhk1 wrote: post #45
th0ma5 wrote: post #44

Earlier quoted context omitted.

The way it is brought up in arguments about how if what you're doing in Julia is slow you can put in the ASM directly in arguments leaves me with the impression that is nonetheless a core part of the "faster than " claims at least. And that's a cop out.

If I remember correctly, this came up originally in the context of comparing heavily optimized Julia code to C code that had inline assembly, in which a statement was made that Julia was obviously slower than C because the C code had hand written assembly in it. Julia, like C, sometimes needs hand-coded assembly to achieve maximal speed. 80% of a fast programming language is not having semantics that are fundamentall…

Yeah I guess that illustrates how everyone is talking about different things, I just think Julia shouldn't use that to claim they are not as slow as many other things, or other vague promises of high level abilities that can somehow traverse paradigms to become the right thing to do at the low level. The fact that you've similarly called out specific caveats to the argument further illustrates how... In my opinion... It is just simply intellectually dishonest how the Julia documentation categorizes others and how promoters of the language don't really know what it means to say that Julia is fast, it clearly can be made not to be.

Re: Geospatial data science with Julia

#47

Post by adgjlsfhk1 »

th0ma5 wrote: post #46
adgjlsfhk1 wrote: post #45

Earlier quoted context omitted.

If I remember correctly, this came up originally in the context of comparing heavily optimized Julia code to C code that had inline assembly, in which a statement was made that Julia was obviously slower than C because the C code had hand written assembly in it. Julia, like C, sometimes needs hand-coded assembly to achieve maximal speed. 80% of a fast programming language is not having semantics that are fundamentall…

Yeah I guess that illustrates how everyone is talking about different things, I just think Julia shouldn't use that to claim they are not as slow as many other things, or other vague promises of high level abilities that can somehow traverse paradigms to become the right thing to do at the low level. The fact that you've similarly called out specific caveats to the argument further illustrates how... In my opinion...…

You realize that the specific things I called out are things julia doesn't do right? Those are the things in Java and Python and most other high level languages that prevent them from being fast. Julia's semantics specifically don't do those things.

Re: Geospatial data science with Julia

#48

Post by th0ma5 »

adgjlsfhk1 wrote: post #47
th0ma5 wrote: post #46

Earlier quoted context omitted.

Yeah I guess that illustrates how everyone is talking about different things, I just think Julia shouldn't use that to claim they are not as slow as many other things, or other vague promises of high level abilities that can somehow traverse paradigms to become the right thing to do at the low level. The fact that you've similarly called out specific caveats to the argument further illustrates how... In my opinion...…

You realize that the specific things I called out are things julia doesn't do right? Those are the things in Java and Python and most other high level languages that prevent them from being fast. Julia's semantics specifically don't do those things.

Honestly I don't know why you're replying, it is not really addressing what I'm talking about and using a lot of the argument styles I'm complaining about.

Re: Geospatial data science with Julia

#49

Post by n4r9 »

BrandonS113 wrote: post #34
n4r9 wrote: post #21

Earlier quoted context omitted.

Is this what you mean by nlopt? https://github.com/stevengj/nlopt If so, it looks like you're interfacing from R to high-performing code written in C. Isn't that exactly what OP was describing?

no, the function it calls is pure R and that is where the the code spends all its time.

Interesting. Have you published the code and/or benchmarks anywhere? This flies against everything I've read about Julia and R.

Re: Geospatial data science with Julia

#50

Post by BrandonS113 »

n4r9 wrote: post #49
BrandonS113 wrote: post #34

Earlier quoted context omitted.

no, the function it calls is pure R and that is where the the code spends all its time.

Interesting. Have you published the code and/or benchmarks anywhere? This flies against everything I've read about Julia and R.

No, its 4 lines of code. I just benchmarked for myself. All it is for 2 vectors x and y of average length 50, and 5 parameters, with exponential, addition, and multiplication, and ultimately sum to return to the optimizer. I was also surprised as I expected c to be much faster. And with Rccp, its actually slower than the R, overhead I guess. When I looked into it, apparently R has really fast code for such vector calculations. With julia, admittingly I did not use simd which would likely make it faster.

Now, I generally use Julia for heavy computes, and usually its much faster than R. But not always.

And this little bit of code runs for hours on the largest instance on AWS every day. Why I was looking so speed it up.

Post reply on HN