how do you compare with Kachelmann and kachelmannwetter.com in terms of data?
For visualization, the UI there is great for a trained eye only, yours seems more accessible. Congrats on that point!
Show HN: Weather API for non-commercial use
21–30 of 154 posts
Re: Show HN: Weather API for non-commercial use
#22Thats cool, I'll be having a closer look. I'm working on an xbar plugin app [1][2] that alerts me when there is a warmer day in the coming week. I used VisualCrossing [3] which seems quite good, and crucially allows 2 weeks historical data, which I think is quite rare. It looks like you offer 2? Darksky did not offer this feature I don't think. For my use the accuracy is not critical. [1] https://github.com/mattarder…
For now I only offer 2 days of past weather data. Most likely I can store some basic weather variables like temperature, humidity, wind, precipitation and weather code for a couple of weeks. I will keep you request in my backlog: https://github.com/open-meteo/open-meteo/issues/27 Thanks!
Great! I'll take a closer look at the API soon
Thanks
Re: Show HN: Weather API for non-commercial use
#23Earlier quoted context omitted.
There is no funding. Costs are negligible. Currently it is running on a cheap rented VM. Per single VM, it should scale to 3+ million API calls per days easily. To keep it simple, I also do not want to have any income with it. In case larger API consumers are on the horizon, I will contact them and ask to sponsor a couple more VMs.
Can you share who your vendor/host is? In another comment you mentioned that you're mmap'ing files on SSDs... on vendors like GCP that sort of hardware (especially the storage volume you'd need to have access to a handful of the local/global models, as you reference on your website) isn't particularly cheap if you want to have 100% uptime. A cheaper/scalable approach instead would be to re-process your data into an a…
The host is netcup in Germany. I use regular KVMs.
Yes, the amount of storage can be an issue, but I want to stay below 500GB of hot-data. One bottleneck is network traffic to copy updated files after each weather model update.
My binary files are just plain Float16 files without an meta data. Logically they similar to 3D Zarr files. I know exactly which bytes I have to read and the kernel cache helps a look to keep it fast.
In theory this data does not have to be a file on SSD. I could also use a block storage directly or request data from S3 via range requests.
One hopelessly over-engineered approach would be to use an `AWS EBS io2 Block Express Volume` use `Multi-Attach` and spawn up to 16 API instances to serve data from it.
Re: Show HN: Weather API for non-commercial use
#24how do you compare with Kachelmann and kachelmannwetter.com in terms of data? For visualization, the UI there is great for a trained eye only, yours seems more accessible. Congrats on that point!
Kachelmann seems to be more media oriented. I am not aware that there is a free API offer.
It looks like Kachelmann is also integrating weather forecasts from different national weather services and the paid version from ECMWF. I am sure the "HD" forecasts mainly use DWD ICON models as well. In this case, data quality would be the same.
I also spend some time to carefully integrate local and global weather models. For solar radiation forecasts, a clear sky radiation model is used to correctly interpolate data to 1h resolution. In the end it is a tradeoff between simple API with low operation cost and data quality/amount.
Re: Show HN: Weather API for non-commercial use
#25I've investigated and tried several (free) weather API's for my web app: https://uw.leftium.com . Open-Meteo looks pretty good! The only thing that seems to be missing for me is precipitation probability. The weathercode is sort of a proxy for this... I'm also interested in sunrise/sunset times; direct_radiation is kind of a proxy for this. Kudos for providing optional historical data with the same API call for the f…
Nice app! Could you please include Celsius as option?
Re: Show HN: Weather API for non-commercial use
#26Looks really awesome! Very quick responses.
I would be interested in seeing the implementation of the service, interesting choice going with Swift. I'm guessing your using something like Vapor for hosting the API?
How are the files designed? I'm guessing you have some cheap way of mapping (lat, long) into a file location? Maybe just by truncating the coordinates to your precision and mapping them onto the file? Using some fancy-pants Uber Hexagons[0]. How is the forecast delivered?
Hmm! Many questions :-). I've been thinking lately of similar (but not for weather) "geographical" API's, and how to store/query data cheaply, so this interested me :-)
Re: Show HN: Weather API for non-commercial use
#27Looks really awesome! Very quick responses. I would be interested in seeing the implementation of the service, interesting choice going with Swift. I'm guessing your using something like Vapor for hosting the API? How are the files designed? I'm guessing you have some cheap way of mapping (lat, long) into a file location? Maybe just by truncating the coordinates to your precision and mapping them onto the file? Using…
Yes, the API part is using Vapor. But there is not much code that actually relies on Vapor. It is mostly doing request parameter parsing and routing.
I use simple 2D lat lon grids for different regions and resolutions. E.g. the global grid uses 2879x1441 pixel grid. The third dimension is time. All data are stored directly as floating points on disk. Because I know the dimensions of the grid, I can read data exactly at the correct position on disk. I use Float16 as well, which saves me 50% disk space compared to Float32.
Fancy hexagons like H3 are not necessary. They could reduce storage by ~30%, but require much more effort and I have to "modify" the forecast. I keep forecast data from national weather services as untouched as possible.
Re: Show HN: Weather API for non-commercial use
#28Looks nice! I love seeing more free options in this space. I'll be keeping an eye out for support in Home Assistant[1] :)
Re: Show HN: Weather API for non-commercial use
#29This is great, I'd love to have dive visibility predictions for my webapp based on location: https://divedb.net/
The challenge I've had is the same, the data format that a lot of these sources provide is in "academic" formats like GRIB, NetCDF etc.. and I've spent a not-insignificant amount of time trying to work through all these formats and what they represent.
Re: Show HN: Weather API for non-commercial use
#30> All data is offered for non-commercial use. With speedy APIs, all data can be served by just a couple of virtual machines for less than a coffee a day. Very impressive! What does your backend stack look like? Are you using any caching or does every API call hit the binary file? > The project went live 2 weeks ago and is slowly being used. What kind of traffic are you getting? Looking forward to future updates!
The API backend is written in Swift with some code in C (conversion Float16, interpolations, some solar radiation libraries). Data are read directly via mmap from local NVMe SSD disks. To scale, I can add new VMs and copy my binary files. There is no frontend cache and I want to keep it this way. Next steps include point-of-presence API virtual machines in different counties. Via GeoDNS this is even faster than a CDN…
What would it take for it to also be offered on a commercial-use basis?