Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geospatial data science with Julia

juliaearth.github.io

Post reply on HN
Print view View original

51–55 of 55 posts

Re: Geospatial data science with Julia

#51

Post by wodenokoto »

BrandonS113 wrote: post #18
wodenokoto wrote: post #11

Earlier quoted context omitted.

Python and R do not generate high performing code. At best they generate calls to high performing code.

I used to think so, but I have a function that gets called about a billion times each and every day as new data comes in, and and takes about 0.01 seconds to evaluate (optimizaiton with nlopt). I tried to code it in c (30% speed improvement) python (twice as slow), Julia (about the same speed). Reason is that call has 5 parameters that operate on a vector of length 50 to return a value to minimize. Turns out R is pre…

I am willing to concede, but also willing to argue that vector and data frame manipulations in R are calls to optimized code.

Like, R is “what if we made a lisp inspired version of Python built around numpy and pandas and then reversed timed”

Re: Geospatial data science with Julia

#52

Post by Nilshg »

BrandonS113 wrote: post #50
n4r9 wrote: post #49

Earlier quoted context omitted.

Interesting. Have you published the code and/or benchmarks anywhere? This flies against everything I've read about Julia and R.

No, its 4 lines of code. I just benchmarked for myself. All it is for 2 vectors x and y of average length 50, and 5 parameters, with exponential, addition, and multiplication, and ultimately sum to return to the optimizer. I was also surprised as I expected c to be much faster. And with Rccp, its actually slower than the R, overhead I guess. When I looked into it, apparently R has really fast code for such vector cal…

If possible I would encourage you to make an MWE and post it on the Julia Discourse - "Julia slower than R for optimization problem" or something like that, there's a good chance that the community will be able to eke out some more performance. Alternatively you might have hit on a case in which Julia itself is currently leaving performance on the table, which would still be helpful for the community to know as being slow is often considered a bug in Julia world.

Re: Geospatial data science with Julia

#53

Post by BrandonS113 »

Nilshg wrote: post #52
BrandonS113 wrote: post #50

Earlier quoted context omitted.

No, its 4 lines of code. I just benchmarked for myself. All it is for 2 vectors x and y of average length 50, and 5 parameters, with exponential, addition, and multiplication, and ultimately sum to return to the optimizer. I was also surprised as I expected c to be much faster. And with Rccp, its actually slower than the R, overhead I guess. When I looked into it, apparently R has really fast code for such vector cal…

If possible I would encourage you to make an MWE and post it on the Julia Discourse - "Julia slower than R for optimization problem" or something like that, there's a good chance that the community will be able to eke out some more performance. Alternatively you might have hit on a case in which Julia itself is currently leaving performance on the table, which would still be helpful for the community to know as being…

Great idea, I do lurk there, just might do that next time I look at the code. My hunch is that simd is the low hanging fruit julia brings. But, and I am not an expert, both might end up doing BLAS anyways, which is why they are so similar.

Re: Geospatial data science with Julia

#54

Post by BrandonS113 »

wodenokoto wrote: post #51
BrandonS113 wrote: post #18

Earlier quoted context omitted.

I used to think so, but I have a function that gets called about a billion times each and every day as new data comes in, and and takes about 0.01 seconds to evaluate (optimizaiton with nlopt). I tried to code it in c (30% speed improvement) python (twice as slow), Julia (about the same speed). Reason is that call has 5 parameters that operate on a vector of length 50 to return a value to minimize. Turns out R is pre…

I am willing to concede, but also willing to argue that vector and data frame manipulations in R are calls to optimized code. Like, R is “what if we made a lisp inspired version of Python built around numpy and pandas and then reversed timed”

I think that is exactly what is happening. Most of my code is much much faster in Julia, and the code is nicer. But R has its moments. Which is good since this particular app has 3K lines, and I do not want to port it to Julia.

And data.tables in R is faster (and I think nicer to write) than DataFrames in Julia. And since data.tables feed my optimization, R still wins.

Re: Geospatial data science with Julia

#55

Post by BrandonS113 »

ekianjo wrote: post #15
beeburrt wrote: post #10

In the preface you list: - Generate high-performance code - Specialize on multiple arguments - Evaluate code interactively - Exploit parallel hardware > This list of requirements eliminates Python, R and other mainstream languages used for data science. Can you elaborate on why/how? Awesome work by the way

R can exploit parallel hardware just fine with Parallel, Future and other libraries like Mirai. The problem is that execution speed is going to be a bottleneck for anything large and when you reach some optimizations, maybe R is not the best language to do the job. But it depends a lot on the use case.

I much prefer parallel in R with mclapply() to the Julia implementation of parallel. One of the few areas where I prefer R to julia (other being R data.tables to julia dataframes)
Post reply on HN