> To make processing this massive dataset practical, we built a Julia pipeline to extract the bits directly into a DuckDB database. The raw data is a bit more than 1GB per annum. The data of interest is 176 bits every 12.5 minutes for 19 years. That is, about 17MB of data. Possibly multiplied by the number of satellites, roughly thirty. It's not big data.
Though I take your point that it’s not big data by the conventional use (i.e. requiring a distributed computing to process). The phrasing in the original article was better: “To make iterative analysis practical, we wrote a Julia pipeline: NetCDF source files are converted to Apache Arrow, then thread-parallel bit extraction is performed into a DuckDB database.”