Live data from Hacker News

Pirate Weather: A free, open, and documented forecast API

pirateweather.net

61–70 of 201 posts

Re: Pirate Weather: A free, open, and documented forecast API

#61
post #37

Earlier quoted context omitted.

Not on the receiving side https://docs.aws.amazon.com/ses/latest/dg/receiving-email-co...

What do you expect SES to do with your mail after receiving it? S3 and Lambda are optional delivery locations, amongst other choices.

Exactly this. It receives the email, now what? You need to run some code on it and so the way to do that is one of the compute services. AWS isn't forcing you to do anything here.

99.9% of SES users I promise are only sending mail anyway. You aren't forced to have Lambdas or anything else to send mail.

Re: Pirate Weather: A free, open, and documented forecast API

#62
post #16

Earlier quoted context omitted.

Have any nerdy Congresspeople wanted to solve this ridiculous inefficiency in recent years. Seems like a natural "the Europeans kick our ass at forecasting" message would go over well and potentially negate the lobbying from Weather Channel and others with a vested interest in perpetuating this stupid situation.

Yes, Congress passed the Weather Research and Forecasting Innovation Act back in 2017 which not only gave massive funding injection to model development activities, data procurement, and more, but also formalized the new next-generation community modeling initiatives which are focused on tackling the dominance of the EC models in the 2-5 day forecast space. The diagnosis of the problems of the American forecast model…

I disagree that my (extremely broad) diagnosis of the problems with forecasting in America is based on flawed premises. You've provided thorough, correct, and important details here but my comment was aimed at the broader HN audience, not on writing the central argument for a discussion on the historical timeline. I think what I said, "Horrible oversight by the federal govt (read, congress) of our technical/scientific forecasting resources means that our forecasting ability is extremely fragmented and poorly organized." is a very concise summary of the things you've laid out here.

As for the details in your comments, the only thing I disagree with strongly is the comment about investment in computing. While sufficient computing resources are central to good forecasting, the lack of investment by NOAA in computing (I sit at NOAA) is a red herring. ECMWF is significantly better than either of the two available American forecasts because they are just better at what they do, all around. In particular with respect to data assimilation. I've sat at meetings with ECMWF forecasters who have asked for access to my in-situ data products and their pipeline is as simple as "point us to the data please". Their data assimilation pipeline is so much more sophisticated and thorough that catching up on that alone would be a huge huge leap. Mind you, not just '4DVAR' the methodology, but literally the way that the community finds and integrates observational data.

ECMWF, the organization, is quite literally structured to strictly accomplish the goal of 'improve the forecast'. Whereas, again broadly, the American institutions are much more a congolemerate of associated researchers doing individual science projects while small teams work on specific segments of the forecast. Yes, we are attempting to fix this. No, we haven't fixed it yet.

This is not to say I don't think we should fund computing or that computing won't help. But we are quite literally 5-10 years of research behind on multiple fronts.

Re: Pirate Weather: A free, open, and documented forecast API

#63

I usually use forecase.weather.gov They provide a nice textual weather page, with English, for next 7 or 8 days [1]. But it is tiring to open 3 or 4 bookmarks, so I thought to consume that as json & make a single page html app with ajax requests. Its still in progress, I spent few minutes yesterday. The surprising thing was (I was ready to fetch the html text & parse it, difficult), I simply changed one parameter to…

That is cool the web front end exposes itself as JSON as well, good tip.

FWIW, they also have a pretty decent API. Its based around zones though, which you'd need to look up. So from that lat and lon, you'd get the zone from:

https://api.weather.gov/points/37.7827,-120.38

Using the zone information, you can get to a forecast:

https://api.weather.gov/gridpoints/STO/72,24/forecast

If you're wanting the current observations, you'd pick a station for that grid such as MOUC1 and go to its observations/latest endpoint:

https://api.weather.gov/stations/MOUC1/observations/latest

The API is in a JSON-LD format so its got a lot of links to related topics in the actual JSON payload. Looking at the JSON can make it somewhat easy to feel out what you need. The documentation is here:

https://www.weather.gov/documentation/services-web-api

Re: Pirate Weather: A free, open, and documented forecast API

#64
post #15

The tech diagram on AWS is insane. This is what I call vendor lock-in porn I'm wondering how much your monthly bill is?

This is a pretty standard looking diagram. It probably isn't very expensive to run.

AWS diagrams are pretty intimidating until you've built a few things with several AWS services.

The benefit is that a lot of maintenance work is taken care of for you, and your costs can be low if you don't need a lot of compute.

Re: Pirate Weather: A free, open, and documented forecast API

#66

I’ve used https://open-meteo.com/ before and I think it’s the same type of open data being exposed. These types of projects are great for stuff like home automation. I’m using to to improve my predictions for power generation (PV) and consumption (heat pump). Planning to is ergst to optimize home battery charging in the future. (Disclaimer; open sourced a small go library for open meteo, but otherwise not affiliated)

Hi, creator of open-meteo.com here! I am using a more wide range of weather models to better cover Europe, Northern Africa and Asia. North America is covered as well with GFS+HRRR and even weather models from the Canadian weather service.

In contrast to pirate weather, I am using compressed local files to more easily run API nodes, without getting a huge AWS bill. Compression is especially important for large historical weather datasets like ERA5 or the 10 km version ERA5-Land.

Let me know if you have any questions!

Re: Pirate Weather: A free, open, and documented forecast API

#67

How does this differ from NOAA weather.gov api? https://www.weather.gov/documentation/services-web-api Or is this a friendlier overlay of their interface?

Good question! It's the same underlying models, but three key differences: 1. Pirate Weather returns data following the Dark Sky syntax, as opposed to the custom NWS format. 2. Pirate Weather has global coverage, the NWS API is only for the US. 3. The NWS one uses human forecasters to come up with their forecasts, compared to Pirate Weather's use of raw model results.

Only on HN would the NWS format be described as 'custom'.

Re: Pirate Weather: A free, open, and documented forecast API

#68
post #11

> Why "PirateWeather"? I've always thought that the HRRR model was pronounced the same way as the classic pirate "ARRR". Also, there is one company out there that thinks APIs can be copyrighted, which might apply here. This answers the first question I had upon seeing the name, which was, is it free, open, documented and legal ? Based on the above the answer seems to be "probably". Very commendable effort, and I hope…

> is it free, open, documented and legal ? I think it says something that we live in a world where NOAA data could be seen as an underground or less than legal means of getting the weather. Perhaps AccuWeather has been successful in their campaign to keep free weather data as obscure as possible.

I think it's more about it having "pirate" in the name that makes one question its legality.

Re: Pirate Weather: A free, open, and documented forecast API

#69
post #5

I was really hoping https://merrysky.net/ had a graphical forecast similar to weather underground's 10 day. I've yet to find anything else that quickly shows everything you'd want to know in a single image. Even the mouseover timeline is perfection. It is so good and seemingly exclusive to WU that I sometimes wonder if they hold a patent for it.

The _presentation_ is impressive, but the forecast is often the raw output from various models, and thus not entirely reliable. For example, I live in a valley where most forecast models smooth over giving more snow/cold than we ever get.

I have a similar issue living on a hill with the windy.com forecasts. Have you had success finding any forecast source/app that’s accurate for your microclimate? Everyone always points to forecastadvisor.com but that’s always pulling from some airport 20 miles away for me totally irrelevant.

Re: Pirate Weather: A free, open, and documented forecast API

#70
post #5

I was really hoping https://merrysky.net/ had a graphical forecast similar to weather underground's 10 day. I've yet to find anything else that quickly shows everything you'd want to know in a single image. Even the mouseover timeline is perfection. It is so good and seemingly exclusive to WU that I sometimes wonder if they hold a patent for it.

"... similar to weather underground's 10 day ..."

Came here, like I come to every weather API/tool discussion, to ask the same thing ... I would really like a less spammy, less bloated 10-day forecast a la WU.

This comes close, however - there's a nice 7-day lookahead ... is it possible that WU is just fudging days 8-9-10 and no real data is available beyond 7 days ?

Post reply on HN