Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geospatial data science with Julia

juliaearth.github.io

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Geospatial data science with Julia

#31

Post by ZeroCool2u »

juliohm wrote: post #25
nraynaud wrote: post #22

Earlier quoted context omitted.

Be mindful that most of julia's geometry code is a wrapper of libGEOS (C version) and libGDAL, that means that you can't easy extend the algorithms, everythig is behind a black box on the C side. Source: I have worked in the field last year, I have a small patch in LibGEOS.jl .

This is not true. Please read the book.

The book is quite interesting, but it does seem like a lot of the underlying work is farmed out to GeoStats.jl, which doesn't really seem to use the same vocabulary I'd expect in other languages using PostGIS or Geopandas etc. For example, I don't see many mentions of Polygons or MultiPolygons when I search. However, I do find this page[1] which seems to define similar(?) equivalents. Can I expect equivalent geospatial joins/queries to be available? I don't see many mentions of the types that I would normally do, especially overlay operations[2].

[1]: https://juliaearth.github.io/GeoStatsDocs/stable/domains.htm...

[2]: https://geopandas.org/en/stable/docs/user_guide/set_operatio...

Re: Geospatial data science with Julia

#32

Post by nraynaud »

juliohm wrote: post #30
nraynaud wrote: post #27

Earlier quoted context omitted.

Scanning the site see mostly points algorithms, the only mention of polygons is a textbook LibGEOS call, I see no network at all. And I see no smart manipulation of anything else than points, I see no subdivision of space, etc.

You probably need to re-scan the book. Meshes.jl is the submodule of the project entirely written in Julia with geometric processing algorithms.

I have worked with it. It was just stating, very little useful code in it. Going back to the source code, I see they added a bit more. A quick look around suggest that only one algorithm uses an indexing structure. Clipping seems limited between a convex polygon and a concave one.

Re: Geospatial data science with Julia

#33

Post by cbkeller »

nraynaud wrote: post #22
alekseiprokopev wrote: post #9

Nice thing about Julia is that you randomly find cool projects like this.

Be mindful that most of julia's geometry code is a wrapper of libGEOS (C version) and libGDAL, that means that you can't easy extend the algorithms, everythig is behind a black box on the C side. Source: I have worked in the field last year, I have a small patch in LibGEOS.jl .

So in other Julia geometry-related projects that may be true, but for this particular corner of the ecosystem the main author (Júlio Hoffimann) has actually implemented much of the underlying geometry and other code from scratch (to the best of my understanding) in pure Julia in a whole set of packages, including e.g.

https://github.com/JuliaGeometry/Meshes.jl https://github.com/JuliaGeometry/Rotations.jl https://github.com/JuliaEarth/GeoStatsBase.jl https://github.com/JuliaEarth/PointPatterns.jl

Re: Geospatial data science with Julia

#34

Post by BrandonS113 »

n4r9 wrote: post #21
BrandonS113 wrote: post #18

Earlier quoted context omitted.

I used to think so, but I have a function that gets called about a billion times each and every day as new data comes in, and and takes about 0.01 seconds to evaluate (optimizaiton with nlopt). I tried to code it in c (30% speed improvement) python (twice as slow), Julia (about the same speed). Reason is that call has 5 parameters that operate on a vector of length 50 to return a value to minimize. Turns out R is pre…

Is this what you mean by nlopt? https://github.com/stevengj/nlopt If so, it looks like you're interfacing from R to high-performing code written in C. Isn't that exactly what OP was describing?

no, the function it calls is pure R and that is where the the code spends all its time.

Re: Geospatial data science with Julia

#35

Post by juliohm »

ZeroCool2u wrote: post #31
juliohm wrote: post #25

Earlier quoted context omitted.

This is not true. Please read the book.

The book is quite interesting, but it does seem like a lot of the underlying work is farmed out to GeoStats.jl, which doesn't really seem to use the same vocabulary I'd expect in other languages using PostGIS or Geopandas etc. For example, I don't see many mentions of Polygons or MultiPolygons when I search. However, I do find this page[1] which seems to define similar(?) equivalents. Can I expect equivalent geospati…

Please read the book. It has all the information in it.

Re: Geospatial data science with Julia

#36

Post by juliohm »

cbkeller wrote: post #33
nraynaud wrote: post #22

Earlier quoted context omitted.

Be mindful that most of julia's geometry code is a wrapper of libGEOS (C version) and libGDAL, that means that you can't easy extend the algorithms, everythig is behind a black box on the C side. Source: I have worked in the field last year, I have a small patch in LibGEOS.jl .

So in other Julia geometry-related projects that may be true, but for this particular corner of the ecosystem the main author (Júlio Hoffimann) has actually implemented much of the underlying geometry and other code from scratch (to the best of my understanding) in pure Julia in a whole set of packages, including e.g. https://github.com/JuliaGeometry/Meshes.jl https://github.com/JuliaGeometry/Rotations.jl https://git…

Exactly. It is a huge effort. Thanks for pointing it out @cbkeller.

Re: Geospatial data science with Julia

#37

Post by th0ma5 »

sainez wrote: post #16
th0ma5 wrote: post #14

Earlier quoted context omitted.

Julia is designed to seem to win arguments as best I can tell... If you complain about the need to break abstractions and the lack of general purpose application you're accused of not understanding. When you say it slow they say you can inline assembler, and when you say that's dumb why have a high level language then, they then say well you don't have to it is fast as is and everyone else is slow, and it just devolv…

You can obviously provide the same abstraction with different implementations that yield different performance characteristics. Julia provides the same level of flexibility (if not more) as Python without any of the design decisions which cause Python to be so slow. I fail to see how this is a contentious point.

Yeah this is a great example of what I'm saying. You obviously don't understand Python then.

Re: Geospatial data science with Julia

#38

Post by th0ma5 »

adgjlsfhk1 wrote: post #17
th0ma5 wrote: post #14

Earlier quoted context omitted.

Julia is designed to seem to win arguments as best I can tell... If you complain about the need to break abstractions and the lack of general purpose application you're accused of not understanding. When you say it slow they say you can inline assembler, and when you say that's dumb why have a high level language then, they then say well you don't have to it is fast as is and everyone else is slow, and it just devolv…

when you say Julia is slow, what are you talking about? even without any fancy tricks, normal Julia code is usually the same speed as the equivalent normal C code

Yes this is another great example... If Julia is fast then why do you need to inline assembler ?

Re: Geospatial data science with Julia

#39

Post by adgjlsfhk1 »

th0ma5 wrote: post #38
adgjlsfhk1 wrote: post #17

Earlier quoted context omitted.

when you say Julia is slow, what are you talking about? even without any fancy tricks, normal Julia code is usually the same speed as the equivalent normal C code

Yes this is another great example... If Julia is fast then why do you need to inline assembler ?

For the same reason that C/C++ allow inline assembly? Languages come in roughly 3 speeds. Slow (e.g. python/R), mostly not slow (e.g. Java/Go), and not slow (e.g. C/Rust). If you want actually fast code (e.g. the speed of BLAS/FFTW etc) you need the combination of a not slow language, code generation, and often hand-coded assembly for the most performance critical parts.

Re: Geospatial data science with Julia

#40

Post by th0ma5 »

adgjlsfhk1 wrote: post #39
th0ma5 wrote: post #38

Earlier quoted context omitted.

Yes this is another great example... If Julia is fast then why do you need to inline assembler ?

For the same reason that C/C++ allow inline assembly? Languages come in roughly 3 speeds. Slow (e.g. python/R), mostly not slow (e.g. Java/Go), and not slow (e.g. C/Rust). If you want actually fast code (e.g. the speed of BLAS/FFTW etc) you need the combination of a not slow language, code generation, and often hand-coded assembly for the most performance critical parts.

I noticed you didn't mention Julia explicitly this time because when you outline the abstractions like this it seems silly to claim something about Julia magically solves the position and purpose of these layers. I can write a PySpark job based on a tutorial that would run circles around a single core Julia process that was designed with contradictory requirements. I just don't see how Julia gets away with claiming it solves all of this in the first page of their documentation without a ton of qualifiers... Except to say that is Julia that's what they do, they make bold claims that obfuscate what performance is and where it comes from.
Post reply on HN