Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Weather API for non-commercial use

open-meteo.com

Post reply on HN
Print view View original

41–50 of 154 posts

Re: Show HN: Weather API for non-commercial use

#41

Post by meteo-jeff »

ricksunny wrote: post #30
meteo-jeff wrote: post #10

Earlier quoted context omitted.

The API backend is written in Swift with some code in C (conversion Float16, interpolations, some solar radiation libraries). Data are read directly via mmap from local NVMe SSD disks. To scale, I can add new VMs and copy my binary files. There is no frontend cache and I want to keep it this way. Next steps include point-of-presence API virtual machines in different counties. Via GeoDNS this is even faster than a CDN…

What would it take for it to also be offered on a commercial-use basis?

I do not actively prevent commercial use. I stated a non-commercial license as precaution to prevent all kinds of legal complications.

Mainly I also intent to reach open-source developers to make access to weather dat easier without the need to instruct users to sign-up for an API key somewhere.

Re: Show HN: Weather API for non-commercial use

#43

Post by berkut »

Looks interesting...

I think the titles for the charts might have the lat/long compass directions swapped: it's showing the correct latitude for my location I selected (from preset - Wellington), but it has a 'E' after it, when it should be 'N'. Similarly, the longitude has 'N' after it, when it should have 'E'.

Re: Show HN: Weather API for non-commercial use

#45

Post by modeless »

meteo-jeff wrote: post #27
filleokus wrote: post #26

Looks really awesome! Very quick responses. I would be interested in seeing the implementation of the service, interesting choice going with Swift. I'm guessing your using something like Vapor for hosting the API? How are the files designed? I'm guessing you have some cheap way of mapping (lat, long) into a file location? Maybe just by truncating the coordinates to your precision and mapping them onto the file? Using…

Yes, the API part is using Vapor. But there is not much code that actually relies on Vapor. It is mostly doing request parameter parsing and routing. I use simple 2D lat lon grids for different regions and resolutions. E.g. the global grid uses 2879x1441 pixel grid. The third dimension is time. All data are stored directly as floating points on disk. Because I know the dimensions of the grid, I can read data exactly…

Very cool! Random weird suggestion: you could get very substantial further space savings (2x-16x) by storing the data in a compressed texture format such as ASTC HDR. What makes compressed textures different from other compressed image formats is that they are designed for random access of the compressed data, with fixed size blocks. Compression is lossy but you can choose the compression level to match your data. I expect that weather data is very suitable for compression so you should be able to get a large benefit with few artifacts.

Re: Show HN: Weather API for non-commercial use

#47

Post by meteo-jeff »

berkut wrote: post #43

Looks interesting... I think the titles for the charts might have the lat/long compass directions swapped: it's showing the correct latitude for my location I selected (from preset - Wellington), but it has a 'E' after it, when it should be 'N'. Similarly, the longitude has 'N' after it, when it should have 'E'.

You are right! The coordinates in the graph have the wrong label. I will quickly change it. Thanks!

Re: Show HN: Weather API for non-commercial use

#48

Post by layoric »

Leftium wrote: post #3

I've investigated and tried several (free) weather API's for my web app: https://uw.leftium.com . Open-Meteo looks pretty good! The only thing that seems to be missing for me is precipitation probability. The weathercode is sort of a proxy for this... I'm also interested in sunrise/sunset times; direct_radiation is kind of a proxy for this. Kudos for providing optional historical data with the same API call for the f…

Just as a note regarding extra calls for weather APIs for historical, there is generally a good reason (coming from someone who built a global weather/energy API).

A) Using Numeric Weather Prediction models (NWP) data as 'historical' data is highly dependent on use case. For some application, this is totally fine and the best available data but with more rapid observation satellites becoming quite low latency, using a +2hour NWP forecast as 'historical' could be quite wrong and source of the data should be clear. For a lot of parameters this is the only choice but certain models have different biases for different areas of the world, limiting the use case further unless your backend is willing to apply improvements based on these known biases.

B) A lot of use cases really are only interested in future predictions and they can also store forecasts at +0 and keep a their own record for performance tracking. Yes it is handy for some, its wasteful for others.

C) This separation allows you to separate your infrastructure as keeping even a relatively short historical record at high geospatial/temporal resolution with good query time gets expensive quickly. Eg, satellite data (I dealt with cloud detection) is commonly at 0.5-2km resolution at update every 5-15 minutes you start having to sub sample locations quickly if you want to store even only weeks of data that is low latency available, if you track 1-2M points at 30 minutes even 2 weeks is 1.3B rows ignoring the number of parameters you might be storing. Where as forecast data can be moved to 'cold' (high latency) cheaper storage more quickly since the latest forecast is the more valuable, quickly losing its value over time. This means you can use storage like Redis to store latest forecasts for rapid ingestion and fetching. Storing historical forecast data per model at +N hours/minutes ahead and you start multiplying your historical data costs again.

Massive kudos meteo-jeff for building this service, accessibility of public data can be extremely frustrating and while GRIB/NetCDF make looking at data over an area through time somewhat more manageable, a lot of use cases are just interested in a single point. My contact is in my bio if you ever want to chat, great public service you are doing, well done!

Re: Show HN: Weather API for non-commercial use

#50

Post by meteo-jeff »

xd1936 wrote: post #42

Fantastic service, thanks for your work on it! I always love seeing government weather data democratized and more accessible. Would it be helpful to have a personal weather station data network contributing to your service as well?

Thanks for your offer. Unfortunately I cannot improve the weather forecast quality with a single station.

National weather services collect all kinds of measurements and observations from weather stations, air planes, balloons, satellites and radar to "assimilate" the global state of the atmosphere. Usually this is full of gaps, because only a fraction of the world is covered by measurement stations.

Once hour 0 is assimilated weather models start to calculate how different processes like solar radiation, clouds and winds change this state.

As Open-Meteo only uses the end-product of this chain, personal weather stations cannot be included anymore.

If you want, you could try to use machine learning and Open-Meteo weather forecasts to build your own tailored forecasts. Hopefully this idea will be picked up by someone with instructions on GitHub.

Post reply on HN