Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We reduced our Google Maps API cost

blog.cityflo.com

Post reply on HN
Print view View original

141–150 of 172 posts

Re: How We reduced our Google Maps API cost

#141

Post by fedorareis »

While that seems like a good start it seems like you would want to supplement that approach with the GTFS and GTFS-realtime feeds from transit agencies where available.

Having worked for a company that provided transit software I am well aware that GTFS-realtime data isn’t always accurate (especially the ETAs). But it seems like combining their ETA estimate with yours and the scheduled arrival time would be a reasonable approach. Or if you just started building a model of the average speeds of the bus on each link using the GTFS-r speed and location data you could probably build up a pretty good historical model of the traffic flow between 2 stops at any given point during the day.

Yes, I am aware that scheduled times in transit can be meaningless. There are a lot of factors that contribute to that from traffic to poor schedule planning. They can however be a useful baseline estimate.

Re: How We reduced our Google Maps API cost

#142

Post by AnotherGoodName »

saila wrote: post #137
AnotherGoodName wrote: post #126

I did the same years ago. We were providing realtime suburb data for a fleet of trains. Each train received a GPS coordinate once per minute, we took this and displayed the suburb. So 1440 updates per day per train. For the fleet it was going to be over $100 a day in API costs. We were going to not display suburb data because of cost. In the end I found a creative commons placename database (geonames.org). For placen…

Have you used PostGIS at all? It seems like a good choice for this kind of lookup. https://postgis.net/

PostGIS would be worth it if there was a lot more to do on that project and full respect to that project. I remember looking at it and it felt like it was a sledgehammer for that particular job. You still had to write up a lot of code to import the dataset you had and then extract it again. Not to mention maintenance since you now have a whole dependency chain installed on your server. This was a single engineer project.

The above solution is a copy paste-able set of classes with zero dependencies that will output the nearest place. Sometimes a single purpose solution is perfect and i'm really not kidding when i say i haven't maintained or even looked at it in over 5 years yet it's still running fine as part of a larger application.

Re: How We reduced our Google Maps API cost

#143

Post by manigandham »

DeathArrow wrote: post #35
kijin wrote: post #6

Since they have actual vehicles on the ground, they could collect data on actual travel times under various traffic conditions and apply a bit of ML to predict how long it's going to take under current conditions. No need for any Google API there. But now that their Google bill is only ~$50/day, it might not be worth building their own prediction system.

If ML could predict everything with accuracy, they would be better by predicting lottery, stock prices or Forex pairs. I used an app which predicts public transportation arrival time based on historic data and it mistakes most often than not, sometimes by a lot.

Technically, that's what Google's API is doing anyway, just with much more data, training and resources.

Re: How We reduced our Google Maps API cost

#144

Post by nitrogen »

graton wrote: post #128
bryanrasmussen wrote: post #125

Earlier quoted context omitted.

hmm, not sure, some DA might want to prosecute it as theft of service.

https://arstechnica.com/tech-policy/2020/03/court-violating-... A snippet from the article: A federal court in Washington, DC, has ruled that violating a website's terms of service isn't a crime under the Computer Fraud and Abuse Act, America's primary anti-hacking law. The lawsuit was initiated by a group of academics and journalists with the support of the American Civil Liberties Union.

Presumably there are other laws available, e.g. ones related to stealing cable TV service.

Re: How We reduced our Google Maps API cost

#145

Post by bryanrasmussen »

graton wrote: post #128
bryanrasmussen wrote: post #125

Earlier quoted context omitted.

hmm, not sure, some DA might want to prosecute it as theft of service.

https://arstechnica.com/tech-policy/2020/03/court-violating-... A snippet from the article: A federal court in Washington, DC, has ruled that violating a website's terms of service isn't a crime under the Computer Fraud and Abuse Act, America's primary anti-hacking law. The lawsuit was initiated by a group of academics and journalists with the support of the American Civil Liberties Union.

Sometimes my simplest comments get misunderstood the greatest; if someone were to want to take this to a criminal court it would be idiotic to go for violating terms of service as the reason, especially for the link you provide, but as I said in my post someone might want to try this as theft of services.

Did you think my use of the word services was somehow related to terms thereof, because no. So to quote wikipedia because it was the first that came up when I googled "Theft of services is the legal term for a crime which is committed when a person obtains valuable services — as opposed to goods — by deception, force, threat or other unlawful means, i.e., without lawfully compensating the provider for these services", do you see how someone might argue that changing out the api key could be seen as a form of deception?

I am not saying that I would think it right that someone bring this to criminal court (I figured I better put that out there as even the simplest of comments can be misunderstood, so who knows what several paragraphs together might lead to), I am not saying that they would even win, I am not saying anyone who did it would be doing so for the purest of motives. But I am saying it does seem something like theft of services by using deception.

on edit: the theft and new api keys refers several ancestors back to this anecdote "Their strategy was to have a pool of API keys attached to new accounts that would take advantage of the Google Maps API free tier, and monitor its usage. As the free tier usage would run out, the system would roll over to a new API key automatically."

Re: How We reduced our Google Maps API cost

#146

Post by eruci »

AnotherGoodName wrote: post #126

I did the same years ago. We were providing realtime suburb data for a fleet of trains. Each train received a GPS coordinate once per minute, we took this and displayed the suburb. So 1440 updates per day per train. For the fleet it was going to be over $100 a day in API costs. We were going to not display suburb data because of cost. In the end I found a creative commons placename database (geonames.org). For placen…

I built a similar open service at https://3geonames.org/api . It supports XML and Json output and is entirely free. Eg: https://api.3geonames.org/51.4647,0.0079.json

Re: How We reduced our Google Maps API cost

#147

Post by CamperBob2 »

mkchoi212 wrote: post #131

Random thought but I think it’s funny how about 40 years ago, people would write about their tricks to save program memory usage. Now, people are talking about how to shave costs of doing API calls.

Meanwhile, the old maxim "All programming is ultimately an exercise in caching" seems to have been forgotten entirely. This whole thread is just surreal.

Re: How We reduced our Google Maps API cost

#148

Post by FireBeyond »

throwaway2048 wrote: post #111
thejosh wrote: post #92

Earlier quoted context omitted.

If you carry lots of android devices it will alter the traffic on Google Maps - https://www.youtube.com/watch?v=k5eL_al_m7Q

I find it insane that the fact that Google is constantly harvesting device locations is blatantly obvious from things like this, but yet if you make that claim in other contexts people will be extremely skeptical and demand extensive proof.

And Apple too - I am constantly seeing this (traffic/congestion flags) on side and residential streets with no traffic monitoring devices in my Apple Maps.

Re: How We reduced our Google Maps API cost

#149

Post by mattlondon »

disillusioned wrote: post #13

Well, that's more clever than a company I did due diligence for. Their strategy was to have a pool of API keys attached to new accounts that would take advantage of the Google Maps API free tier, and monitor its usage. As the free tier usage would run out, the system would roll over to a new API key automatically. Wrote that one up in big red marker in my report...

From what I understand (I have no insider info/hints or anything related), this sort of thing was precisely why Google radically changed the pricing structure to make the free tier much smaller for Maps API recently.

I.e. Google knew there was rampant abuse by people like this (this example is not the only thing like this I have heard of...) so Google fixed the glitch and in the process ruined it for all the people genuinely using the service's free tier.

This is why we can't have nice things :) I guess we are lucky that Google didn't decide to just cut their losses and close the whole shebang down - that would be a shame as Google maps is really useful IMO.

Re: How We reduced our Google Maps API cost

#150

Post by jdofaz »

Danieru wrote: post #116
throwaway2048 wrote: post #111

Earlier quoted context omitted.

I find it insane that the fact that Google is constantly harvesting device locations is blatantly obvious from things like this, but yet if you make that claim in other contexts people will be extremely skeptical and demand extensive proof.

Those phones all had Google Maps app open. That Google is harvesting device locations while you have a Google app open showing your GPS location should not be a surprise. What would be a surprise is if Google forced on your GPS to gather your location in other contexts.

I have to keep location services (android) disabled to keep my daily movement from appearing in google location history.
Post reply on HN