Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geospatial data science with Julia

juliaearth.github.io

Post reply on HN
Print view View original

21–30 of 55 posts

Re: Geospatial data science with Julia

#21

Post by n4r9 »

BrandonS113 wrote: post #18
wodenokoto wrote: post #11

Earlier quoted context omitted.

Python and R do not generate high performing code. At best they generate calls to high performing code.

I used to think so, but I have a function that gets called about a billion times each and every day as new data comes in, and and takes about 0.01 seconds to evaluate (optimizaiton with nlopt). I tried to code it in c (30% speed improvement) python (twice as slow), Julia (about the same speed). Reason is that call has 5 parameters that operate on a vector of length 50 to return a value to minimize. Turns out R is pre…

Is this what you mean by nlopt? https://github.com/stevengj/nlopt

If so, it looks like you're interfacing from R to high-performing code written in C. Isn't that exactly what OP was describing?

Re: Geospatial data science with Julia

#22

Post by nraynaud »

alekseiprokopev wrote: post #9

Nice thing about Julia is that you randomly find cool projects like this.

Be mindful that most of julia's geometry code is a wrapper of libGEOS (C version) and libGDAL, that means that you can't easy extend the algorithms, everythig is behind a black box on the C side. Source: I have worked in the field last year, I have a small patch in LibGEOS.jl .

Re: Geospatial data science with Julia

#23

Post by fiedzia »

jstrickshire wrote: post #4

I have a passion project 4x4anarchy.com that operates with a Python-MariaDB system for querying map data by latitude and longitude, transforming it into GeoJSON for map display. The website deals with sizable tables, approximately 1 GB in size. I've made extensive optimizations, relying on well-structured indexes, caching mechanisms, and query optimization to enhance performance. Given these circumstances, how might…

If all you do is "find records within x miles from lat,lon", solr/ES is the best solution. I think it can match a shape too.

Re: Geospatial data science with Julia

#24

Post by eigenket »

juliohm wrote: post #2

Geospatial Data Science with Julia presents a fresh approach to data science with geospatial data and the Julia programming language. It contains best practices for writing clean, readable and performant code in geoscientific applications involving sophisticated representations of the (sub)surface of the Earth such as unstructured meshes made of 2D and 3D geometries.

Are you a bot? Why did you copy and paste the top paragraph of the linked page?

Re: Geospatial data science with Julia

#25

Post by juliohm »

nraynaud wrote: post #22
alekseiprokopev wrote: post #9

Nice thing about Julia is that you randomly find cool projects like this.

Be mindful that most of julia's geometry code is a wrapper of libGEOS (C version) and libGDAL, that means that you can't easy extend the algorithms, everythig is behind a black box on the C side. Source: I have worked in the field last year, I have a small patch in LibGEOS.jl .

This is not true. Please read the book.

Re: Geospatial data science with Julia

#26

Post by jstrickshire »

mustknow2201 wrote: post #20
jstrickshire wrote: post #4

I have a passion project 4x4anarchy.com that operates with a Python-MariaDB system for querying map data by latitude and longitude, transforming it into GeoJSON for map display. The website deals with sizable tables, approximately 1 GB in size. I've made extensive optimizations, relying on well-structured indexes, caching mechanisms, and query optimization to enhance performance. Given these circumstances, how might…

I know you said it's a passion project, but you should probably still give the correct OSM attribution https://osmfoundation.org/wiki/Licence/Attribution_Guideline...

I appreciate you calling that out, I will get that done.

Re: Geospatial data science with Julia

#27

Post by nraynaud »

juliohm wrote: post #25
nraynaud wrote: post #22

Earlier quoted context omitted.

Be mindful that most of julia's geometry code is a wrapper of libGEOS (C version) and libGDAL, that means that you can't easy extend the algorithms, everythig is behind a black box on the C side. Source: I have worked in the field last year, I have a small patch in LibGEOS.jl .

This is not true. Please read the book.

Scanning the site see mostly points algorithms, the only mention of polygons is a textbook LibGEOS call, I see no network at all. And I see no smart manipulation of anything else than points, I see no subdivision of space, etc.

Re: Geospatial data science with Julia

#28

Post by qfwfq_ »

gabegm wrote: post #19
jstrickshire wrote: post #4

I have a passion project 4x4anarchy.com that operates with a Python-MariaDB system for querying map data by latitude and longitude, transforming it into GeoJSON for map display. The website deals with sizable tables, approximately 1 GB in size. I've made extensive optimizations, relying on well-structured indexes, caching mechanisms, and query optimization to enhance performance. Given these circumstances, how might…

It would depend on where most of the processing is happening. PostGIS gives you the benefit of spatial indexes which are extremely performant. I've seen Python GeoSpatial applications taking hours to finish processing which only took a few minutes when shifted onto PostGIS. If you're also doing a lot of processing in Python, exploring other languages could also help. In the case of Julia you get a typed language that…

Geopandas has had spatial indexing available for quite a long time...

https://geopandas.org/en/stable/docs/reference/sindex.html

I think that the challenge for most is that the PostGIS query planner does the indexing for you in most queries, while a naive all-pairs comparison in geopandas/shapely won't tell you to use the .sindex attribute instead.

Re: Geospatial data science with Julia

#29

Post by forgotpwd16 »

eigenket wrote: post #24
juliohm wrote: post #2

Geospatial Data Science with Julia presents a fresh approach to data science with geospatial data and the Julia programming language. It contains best practices for writing clean, readable and performant code in geoscientific applications involving sophisticated representations of the (sub)surface of the Earth such as unstructured meshes made of 2D and 3D geometries.

Are you a bot? Why did you copy and paste the top paragraph of the linked page?

Seems to be the author and copied it as some form of abstract. @juliohm no need to be doing that.

Re: Geospatial data science with Julia

#30

Post by juliohm »

nraynaud wrote: post #27
juliohm wrote: post #25

Earlier quoted context omitted.

This is not true. Please read the book.

Scanning the site see mostly points algorithms, the only mention of polygons is a textbook LibGEOS call, I see no network at all. And I see no smart manipulation of anything else than points, I see no subdivision of space, etc.

You probably need to re-scan the book. Meshes.jl is the submodule of the project entirely written in Julia with geometric processing algorithms.
Post reply on HN