How We Mapped 1.3M Data Points Using Mapbox • Hacker News Board

Re: How We Mapped 1.3M Data Points Using Mapbox

#11

Post by reaperducer » Thu, Sep 13, 2018, 7:59 PM UTC

dagw wrote: post #9

llao wrote: post #8

Next time they better talk to experts for their own sanities' sake...! > We set the interpolation process running in QGIS on a Mac Pro and, a mere 11 days later, found ourselves the proud owners of a raster 250m2 grid layer Probably done in minutes to hours with a properly setup PostGIS database and one query. "Data Journalists" seem to love it when things take long, thinking they are doing something super innovative…

Yea, as someone who does this sort of thing for a living much of that was quite painful to read. I mean hats off to them for getting a nice result, but they could have done it so much more efficiently by using the right tools.

While I don't disagree, this isn't something they do on a daily basis like you, so they're going in with blinders on.

Unfortunately, mapping documentation is absolutely abysmal. 90% of what's available online is either eight years out of date, riddled with TODO:'s, or a mish-mash of incompatible versions.

As someone who doesn't do this for a living, but has to build 70,000 maps on a weekly basis, I know it's a nightmare. Especially when you're going in for the first time.

I think they did a pretty good job considering they have to be jacks of all trades.

Re: How We Mapped 1.3M Data Points Using Mapbox

#12

Post by reaperducer » Thu, Sep 13, 2018, 8:02 PM UTC

dylrich wrote: post #2

It's interesting seeing the process they went through to work around MapBox's hosting limitations, but the map doesn't feel terribly usable to me. The chart doesn't update on pan/zoom (at least for me in Firefox) and the raster area is a little funky, I wish it was using a standard interpolation that covered everywhere instead of selective areas. Also, I am loading a ton of data on this page. Interactive maps are nic…

>it's totally not usable on a mobile network for a ton of people

As noted in the sections:

4. Maps aren’t truly responsive by default (but are annoying on mobile by default)

5. Performance, particularly on mobile, will need all the help it can get

As someone who worked for me used to say: "It's better than good. It's done."

Re: How We Mapped 1.3M Data Points Using Mapbox

#13

Post by dagw » Thu, Sep 13, 2018, 8:10 PM UTC

reaperducer wrote: post #11

dagw wrote: post #9

Earlier quoted context omitted.

Yea, as someone who does this sort of thing for a living much of that was quite painful to read. I mean hats off to them for getting a nice result, but they could have done it so much more efficiently by using the right tools.

While I don't disagree, this isn't something they do on a daily basis like you, so they're going in with blinders on. Unfortunately, mapping documentation is absolutely abysmal. 90% of what's available online is either eight years out of date, riddled with TODO:'s, or a mish-mash of incompatible versions. As someone who doesn't do this for a living, but has to build 70,000 maps on a weekly basis, I know it's a nightm…

Yea, I guess I was being a bit harsh. They delivered a nice looking result and at the end of day that's what counts. And honestly thinking back at the first several times I did something similar I can't really say I did a more efficient job.

Re: How We Mapped 1.3M Data Points Using Mapbox

#14

Post by brootstrap » Thu, Sep 13, 2018, 8:47 PM UTC

anyone else have experience with GDAL? god i hate that software so much but it does some cool stuff. Been working with GDAL/QGIS on the regs for about 5 years now and it's pretty amazing what you can do. everything looks great in a map

Re: How We Mapped 1.3M Data Points Using Mapbox

#15

Post by llao » Thu, Sep 13, 2018, 8:57 PM UTC

reaperducer wrote: post #11

dagw wrote: post #9

Earlier quoted context omitted.

Yea, as someone who does this sort of thing for a living much of that was quite painful to read. I mean hats off to them for getting a nice result, but they could have done it so much more efficiently by using the right tools.

While I don't disagree, this isn't something they do on a daily basis like you, so they're going in with blinders on. Unfortunately, mapping documentation is absolutely abysmal. 90% of what's available online is either eight years out of date, riddled with TODO:'s, or a mish-mash of incompatible versions. As someone who doesn't do this for a living, but has to build 70,000 maps on a weekly basis, I know it's a nightm…

My OP was very negative, sorry.

Absolutely! Solving what you are set out to do is a great result and I am super happy for them and their end result!

But this state of documentation (as you call it, I would also/rather consider all the needed background knowledge and lingo to be a hurdle) is all the more reason to just document their goal, their data(s), their abilities and capabilities and ask an expert for 15 minutes of their time for input and pointers. That is so much more efficient!

Re: How We Mapped 1.3M Data Points Using Mapbox

#16

Post by llao » Thu, Sep 13, 2018, 9:00 PM UTC

brootstrap wrote: post #14

anyone else have experience with GDAL? god i hate that software so much but it does some cool stuff. Been working with GDAL/QGIS on the regs for about 5 years now and it's pretty amazing what you can do. everything looks great in a map

Yes, ask me anything and I will try to help you.

Can you try to categorize what you hate about it? If it are specific things and not the general hurdle of figuring where goes what and what means what.

Re: How We Mapped 1.3M Data Points Using Mapbox

#17

Post by kaybe » Thu, Sep 13, 2018, 10:21 PM UTC

reaperducer wrote: post #11

dagw wrote: post #9

Earlier quoted context omitted.

Yea, as someone who does this sort of thing for a living much of that was quite painful to read. I mean hats off to them for getting a nice result, but they could have done it so much more efficiently by using the right tools.

While I don't disagree, this isn't something they do on a daily basis like you, so they're going in with blinders on. Unfortunately, mapping documentation is absolutely abysmal. 90% of what's available online is either eight years out of date, riddled with TODO:'s, or a mish-mash of incompatible versions. As someone who doesn't do this for a living, but has to build 70,000 maps on a weekly basis, I know it's a nightm…

I'd be interested in mapping solutions for python. What is your preferred workflow, HN?

Re: How We Mapped 1.3M Data Points Using Mapbox

#18

Post by [deleted] » Fri, Sep 14, 2018, 1:22 AM UTC

kaybe wrote: post #17

reaperducer wrote: post #11

Earlier quoted context omitted.

While I don't disagree, this isn't something they do on a daily basis like you, so they're going in with blinders on. Unfortunately, mapping documentation is absolutely abysmal. 90% of what's available online is either eight years out of date, riddled with TODO:'s, or a mish-mash of incompatible versions. As someone who doesn't do this for a living, but has to build 70,000 maps on a weekly basis, I know it's a nightm…

I'd be interested in mapping solutions for python. What is your preferred workflow, HN?

[deleted]

Re: How We Mapped 1.3M Data Points Using Mapbox

#19

Post by llao » Fri, Sep 14, 2018, 6:42 AM UTC

kaybe wrote: post #17

reaperducer wrote: post #11

Earlier quoted context omitted.

While I don't disagree, this isn't something they do on a daily basis like you, so they're going in with blinders on. Unfortunately, mapping documentation is absolutely abysmal. 90% of what's available online is either eight years out of date, riddled with TODO:'s, or a mish-mash of incompatible versions. As someone who doesn't do this for a living, but has to build 70,000 maps on a weekly basis, I know it's a nightm…

I'd be interested in mapping solutions for python. What is your preferred workflow, HN?

Depends on the kind of data. Check out cartopy, descartes, rasterio, geopandas.

Re: How We Mapped 1.3M Data Points Using Mapbox

#20

Post by dagw » Fri, Sep 14, 2018, 8:01 AM UTC

llao wrote: post #19

kaybe wrote: post #17

Earlier quoted context omitted.

I'd be interested in mapping solutions for python. What is your preferred workflow, HN?

Depends on the kind of data. Check out cartopy, descartes, rasterio, geopandas.

Another general point is that once you're working with sufficiently large data sets you no longer have a GIS problem, but a Big Data/High Performance Computing problem and you need to start working with tools available in those domains together with the GIS tools.

Like in the article. If you have a problem that takes 11 days to run in QGIS, then you shouldn't be using QGIS, but a tool that is designed for processing large amounts of data. 1-2 million points might be a lot from a GIS tools perspective, but is absolutely nothing from a big data/HPC perspective, so be sure to check what those guy are doing.