A journey of optimization of cloud-based geospatial data processing • Hacker News Board

Re: A journey of optimization of cloud-based geospatial data processing

#11

Post by kbgg » Sun, Dec 15, 2024, 5:55 PM UTC

sid_tf wrote: post #9

kbgg wrote: post #4

Earlier quoted context omitted.

Totally agree. When I worked at Radiant Earth the most frequent question we got was "I have this data I want to share, how do I create a good STAC for it?" It was a totally valid question but one that's practically impossible to answer. As a result there's just so much variation between STACs and you never really know what you're going to get.

Hi Sid here one of authors of blog. STAC being open and extensible, makes it a double edge sword yes. Quite refreshing to hear this from someone who was at Radiant coz, it shows we still haven't reached a great way of sharing data yet. What do you think of attempts like Source Cooperative?

There's clearly a need for Source Cooperative given the overwhelming positive feedback we received during the beta. However, Source Cooperative is entirely dependent on Microsoft and Amazon subsidizing all of the S3 / Azure Blob Storage costs. They could pull the rug out at any moment, like we've seen with Planetary Computer, and Source Cooperative would no longer be sustainable.

Disclaimer: I built Source Cooperative and left Radiant 2 months ago.

Re: A journey of optimization of cloud-based geospatial data processing

#12

Post by whinvik » Sun, Dec 15, 2024, 7:16 PM UTC

sid_tf wrote: post #7

whinvik wrote: post #2

As someone who deals with geospatial processes like this daily, I have 2 notes. 1. STAC implementations are already complicated. Not everyone has good catalogues, or that work uniformly well for all types of queries. 2. Using STAC geoparquet on top and then another layer on top would mean we have to self host yet another catalog, essentially a new standard. In short, even though I believe STAC adoption is what we sho…

Agree on both fronts! STAC is pretty complex. My attempt here was to make raw data access easy and fast, not to solve STAC, which I believe stac-geoparquet basically makes an attempt to fix (makes it columnar and hence faster to query at scale). And yes, having a parquet will add overhead of needing some form of catalog. But I believe we are very close to having Iceberg with native geo types being that catalog. at th…

Not sure I understand. The blog mentions adding columns to the Geoparquet so it's either an extension or a new standard.

Not going to discourage what you are doing but just reading the blog, my immediate instinct is to not try to check out what you build.

Re: A journey of optimization of cloud-based geospatial data processing

#13

Post by sid_tf » Mon, Dec 16, 2024, 6:22 AM UTC

whinvik wrote: post #12

sid_tf wrote: post #7

Earlier quoted context omitted.

Agree on both fronts! STAC is pretty complex. My attempt here was to make raw data access easy and fast, not to solve STAC, which I believe stac-geoparquet basically makes an attempt to fix (makes it columnar and hence faster to query at scale). And yes, having a parquet will add overhead of needing some form of catalog. But I believe we are very close to having Iceberg with native geo types being that catalog. at th…

Not sure I understand. The blog mentions adding columns to the Geoparquet so it's either an extension or a new standard. Not going to discourage what you are doing but just reading the blog, my immediate instinct is to not try to check out what you build.

Sure! Glad you shared your honest opinion. But I just want to reiterate that the blog and its subsequent library which will be released is not being done to create a new standard. All throughout the blog its been clearly stated that this is "a new approach", not "new standard" or "new format" or even "better standard".

Re: A journey of optimization of cloud-based geospatial data processing

#14

Post by sid_tf » Mon, Dec 16, 2024, 6:23 AM UTC

lmc wrote: post #10

Interesting ideas. If you do some subsequent post on this, it would be great to see how the perf benefits scale with different sizes of AoIs.

Thanks! I will be posting more on this, especially as we try out more use cases like ML dataset curation, globally distributed time series analysis, and more. Surely will take your feedback into account of having variations in AOI sizes. Here;s the terrafloww linkedin, you can find me on it too, https://www.linkedin.com/company/terrafloww-inc/