Viewing profile — PolarizedPoutin

https://aliramadhan.me/ I recently redesigned my personal website to mimic Earth’s layers and also started blogging. Thanks for putting this together, I love browsing through uniqu…

I just wrote up a small package, matplotloom [1], to simplify and speed up making animations with matplotlib. I've also written some documentation [2]. It's published on PyPI [3] s…

Thank you for reading and for the links! I'm trying out Clickhouse for the next post. Definitely excited for sub 24 hour data loading! I haven't heard of VictoriaMetrics but that's…

The data is publicly available! The data is freely available from the Climate Change Service [1] which has a nice API but download speeds can be a bit slow. You'll have to sign up …

What do you mean by the heat distribution of energy? Do you mean like how much heat is received from the sun at a particular location and particular time? If so, then temperature i…

GraphCast was trained on this exact same data! From https://deepmind.google/discover/blog/graphcast-ai-model-for... > Crucially, GraphCast and traditional approaches go hand-in-han…

The full dataset is quite huge (~9 petabytes and growing) out of which I'm using just ~8 terabytes. The data is freely available from the Climate Change Service [1] which has a nic…

The full dataset is quite huge (~9 petabytes and growing) out of which I'm using just ~8 terabytes. Still quite big to upload. The data is freely available from the Climate Change …

Yeah I was thinking about this and hoped that Postgres had a `float2` data type but `int2` would have to work. I could scale the numbers to fit them into 2 bytes with minimal loss …

It's a good question! It's true that the output is massive, I believe ~9 petabytes and growing. But running the model is super expensive. It runs on ECMWF's supercomputer. Not sure…

I'm hoping to compare TimescaleDB and Clickhouse to see how big the difference is for different queries! My impression is that TimescaleDB gives you some columnar features, but may…

Had a read through parts 1 and 2, thank you for the engaging reads! Love how you've formatted your posts with the margin notes too. Thank you for providing the function to write nu…

Thank you for reading through and for your feedback! Excited to try your settings to disable the WAL and other overhead and see if I get even faster inserts. Also glad to hear an e…

Hoping to find out when I compare TimescaleDB vs. Clickhouse!

Haha simplistic but probably faster and more space-efficient than a relational database. Sounds like rabernat and open-meteo who commented here do something similar to you and find…

Thank you for mentioning paleoclimatology! Not sure if what I'm doing with ERA5 data is that rigorous haha, but one of my favorite plots is https://commons.wikimedia.org/wiki/File:…

The data is definitely sparse in the earlier decades, especially over unpopulated areas and in developing nations, due to the lack of regularly recorded weather observations (senso…

Yes you are correct of course. ERA5 is climate model output constrained to match weather observations, not actual observations, which I do note in the post.

Are you suggesting compressing the data using like Fourier series or wavelets? I know some of the variables have sharp gradients, especially variables like precipitation, which pro…

Thank you! I've relied on that USGS Projects manual multiple times haha. Working with satellite data some of it was in somewhat obscure projections and the manual always told you h…

Thanks for the link on the trillion row challenge, interesting read! I'm looking at queries and indexes next and I'm hoping to include Clickhouse in that comparison.

Thank you for the link! I've mostly used TablePlus (not free I think) and matplotlib (via psycopg3) to plot data returned by queries, but this looks like it'll be faster to use. I'…

Haha thank you for reading and glad you found it engaging! Maybe it's the benefit of being a beginner and not having any skin in the game. I did a lot of searching but couldn't fin…

Thanks for pointing out the status of ERA5 on GCP and AWS. I'll have to try the binning and compare it with listing all the values and finding the 99th percentile! But yeah the spa…

Hey Ryan and thank you for the feedback! I agree that storing the data is appropriately chunked Zarr files is almost surely going to be faster, simpler to set up, and take up less …