Earlier quoted context omitted.
https://www.kaggle.com/datasets/nickalt/hacker-news
this is amazing -- thanks. side note: when you want to get more data you can just get the new data, and save it as hacker-news-part-2.parquet - then when you use duckdb / malloy you can read them as a glob (hacker-news*.parquet) - and it is if you were reading one file.
Great tip. I used this little project as an exercise to some of learn DuckDB and Parquet. Still thinking about the best tradeoff for file format. An extra complicating factor is that items can change after they are posted so to get the most recent data, it's a good idea to re-request at least some range of historical items.